Instruction:
How do I get a list of locally installed Python modules?

How do I get a list of Python modules installed on my computer?

Output: Solution Do not use with pip > 10.0! My 50 cents for getting a pip freeze-like list from a Python script: import pip installed_packages = pip.get_installed_distributions() installed_packages_list = sorted(["%s==%s" % (i.key, i.version) for i in installed_packages]) print(installed_packages_list) As a (too long) one liner: sorted(["%s==%s" % (i.key, i.version) for i in pip.get_installed_distributions()]) Giving: ['behave==1.2.4', 'enum34==1.0', 'flask==0.10.1', 'itsdangerous==0.24', 'jinja2==2.7.2', 'jsonschema==2.3.0', 'markupsafe==0.23', 'nose==1.3.3', 'parse-type==0.3.4', 'parse==1.6.4', 'prettytable==0.7.2', 'requests==2.3.0', 'six==1.6.1', 'vioozer-metadata==0.1', 'vioozer-users-server==0.1', 'werkzeug==0.9.4'] Scope This solution applies to the system scope or to a virtual environment scope, and covers packages installed by setuptools, pip and (god forbid) easy_install. My use case I added the result of this call to my Flask server, so when I call it with http://example.com/exampleServer/environment I get the list of packages installed on the server's virtualenv. It makes debugging a whole lot easier. Caveats I have noticed a strange behaviour of this technique - when the Python interpreter is invoked in the same directory as a setup.py file, it does not list the package installed by setup.py. Steps to reproduce: Create a virtual environment $ cd /tmp $ virtualenv test_env New python executable in test_env/bin/python Installing setuptools, pip...done. $ source test_env/bin/activate (test_env) $ Clone a Git repository with setup.py (test_env) $ git clone https://github.com/behave/behave.git Cloning into 'behave'... remote: Reusing existing pack: 4350, done. remote: Total 4350 (delta 0), reused 0 (delta 0) Receiving objects: 100% (4350/4350), 1.85 MiB | 418.00 KiB/s, done. Resolving deltas: 100% (2388/2388), done. Checking connectivity... done. We have behave's setup.py in /tmp/behave: (test_env) $ ls /tmp/behave/setup.py /tmp/behave/setup.py Install the Python package from the Git repository (test_env) $ cd /tmp/behave && pip install . running install ... Installed /private/tmp/test_env/lib/python2.7/site-packages/enum34-1.0-py2.7.egg Finished processing dependencies for behave==1.2.5a1 If we run the aforementioned solution from /tmp >>> import pip >>> sorted(["%s==%s" % (i.key, i.version) for i in pip.get_installed_distributions()]) ['behave==1.2.5a1', 'enum34==1.0', 'parse-type==0.3.4', 'parse==1.6.4', 'six==1.6.1'] >>> import os >>> os.getcwd() '/private/tmp' If we run the aforementioned solution from /tmp/behave >>> import pip >>> sorted(["%s==%s" % (i.key, i.version) for i in pip.get_installed_distributions()]) ['enum34==1.0', 'parse-type==0.3.4', 'parse==1.6.4', 'six==1.6.1'] >>> import os >>> os.getcwd() '/private/tmp/behave' behave==1.2.5a1 is missing from the second example, because the working directory contains behave's setup.py file. I could not find any reference to this issue in the documentation. Perhaps I shall open a bug for it.


Instruction:
Maximum and Minimum values for ints

How do I represent minimum and maximum values for integers in Python? In Java, we have Integer.MIN_VALUE and Integer.MAX_VALUE. See also: What is the maximum float in Python?.

Output: Python 3 In Python 3, this question doesn't apply. The plain int type is unbounded. However, you might actually be looking for information about the current interpreter's word size, which will be the same as the machine's word size in most cases. That information is still available in Python 3 as sys.maxsize, which is the maximum value representable by a signed word. Equivalently, it's the size of the largest possible list or in-memory sequence. Generally, the maximum value representable by an unsigned word will be sys.maxsize * 2 + 1, and the number of bits in a word will be math.log2(sys.maxsize * 2 + 2). See this answer for more information. Python 2 In Python 2, the maximum value for plain int values is available as sys.maxint: >>> sys.maxint # on my system, 2**63-1 9223372036854775807 You can calculate the minimum value with -sys.maxint - 1 as shown in the docs. Python seamlessly switches from plain to long integers once you exceed this value. So most of the time, you won't need to know it.


Instruction:
How to print a number using commas as thousands separators

How do I print an integer with commas as thousands separators? 1234567 ⟶ 1,234,567 It does not need to be locale-specific to decide between periods and commas.

Output: Locale-agnostic: use _ as the thousand separator f'{value:_}' # For Python ≥3.6 Note that this will NOT format in the user's current locale and will always use _ as the thousand separator, so for example: 1234567 ⟶ 1_234_567 English style: use , as the thousand separator '{:,}'.format(value) # For Python ≥2.7 f'{value:,}' # For Python ≥3.6 Locale-aware import locale locale.setlocale(locale.LC_ALL, '') # Use '' for auto, or force e.g. to 'en_US.UTF-8' '{:n}'.format(value) # For Python ≥2.7 f'{value:n}' # For Python ≥3.6 Reference Per Format Specification Mini-Language, The ',' option signals the use of a comma for a thousands separator. For a locale aware separator, use the 'n' integer presentation type instead. and: The '_' option signals the use of an underscore for a thousands separator for floating point presentation types and for integer presentation type 'd'. For integer presentation types 'b', 'o', 'x', and 'X', underscores will be inserted every 4 digits.


Instruction:
How do I iterate through two lists in parallel?

I have two iterables, and I want to go over them in pairs: foo = [1, 2, 3] bar = [4, 5, 6] for (f, b) in iterate_together(foo, bar): print("f:", f, " | b:", b) That should result in: f: 1 | b: 4 f: 2 | b: 5 f: 3 | b: 6 One way to do it is to iterate over the indices: for i in range(len(foo)): print("f:", foo[i], " | b:", bar[i]) But that seems somewhat unpythonic to me. Is there a better way to do it? Related tasks: * How to merge lists into a list of tuples? - given the above foo and bar, create the list [(1, 4), (2, 5), (3, 6)]. * How can I make a dictionary (dict) from separate lists of keys and values? - create the dict {1: 4, 2: 5, 3: 6}. * Create a dictionary with comprehension - constructing dict using zip in a dict comprehension.

Output: Python 3 for f, b in zip(foo, bar): print(f, b) zip stops when the shorter of foo or bar stops. In Python 3, zip returns an iterator of tuples, like itertools.izip in Python2. To get a list of tuples, use list(zip(foo, bar)). And to zip until both iterators are exhausted, you would use itertools.zip_longest. Python 2 In Python 2, zip returns a list of tuples. This is fine when foo and bar are not massive. If they are both massive then forming zip(foo,bar) is an unnecessarily massive temporary variable, and should be replaced by itertools.izip or itertools.izip_longest, which returns an iterator instead of a list. import itertools for f,b in itertools.izip(foo,bar): print(f,b) for f,b in itertools.izip_longest(foo,bar): print(f,b) izip stops when either foo or bar is exhausted. izip_longest stops when both foo and bar are exhausted. When the shorter iterator(s) are exhausted, izip_longest yields a tuple with None in the position corresponding to that iterator. You can also set a different fillvalue besides None if you wish. See here for the full story. Note also that zip and its zip-like brethen can accept an arbitrary number of iterables as arguments. For example, for num, cheese, color in zip([1,2,3], ['manchego', 'stilton', 'brie'], ['red', 'blue', 'green']): print('{} {} {}'.format(num, color, cheese)) prints 1 red manchego 2 blue stilton 3 green brie


Instruction:
Usage of __slots__?

What is the purpose of __slots__ in Python — especially with respect to when I would want to use it, and when not?

Output: TLDR The special attribute __slots__ allows you to explicitly state which instance attributes you expect your object instances to have, with the expected results: faster attribute access. space savings in memory. The space savings is from: Storing value references in slots instead of __dict__. Denying __dict__ and __weakref__ creation if parent classes deny them and you declare __slots__. This has the effect of denying the creation of non-slotted attributes on its instances, including within the class body (such as in methods like __init__). Quick Caveats Small caveat, you should only declare a particular slot one time in an inheritance tree. For example: class Base: __slots__ = 'foo', 'bar' class Right(Base): __slots__ = 'baz', class Wrong(Base): __slots__ = 'foo', 'bar', 'baz' # redundant foo and bar Python doesn't object when you get this wrong (it probably should), and problems might not otherwise manifest, but your objects will take up more space than they should. Python 3.8: >>> from sys import getsizeof >>> getsizeof(Right()), getsizeof(Wrong()) (56, 72) This is because Base's slot descriptor has a slot separate from Wrong's. This shouldn't usually come up, but it could: >>> w = Wrong() >>> w.foo = 'foo' >>> Base.foo.__get__(w) Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: foo >>> Wrong.foo.__get__(w) 'foo' The biggest caveat is for multiple inheritance - multiple "parent classes with nonempty slots" cannot be combined. To accommodate this restriction, follow best practices: create abstractions with empty __slots__ for every parent class (or for every parent class but one), then inherit from these abstractions instead of their concrete versions in your new concrete class. (The original parent classes should also inherit from their respective abstractions, of course.) See section on multiple inheritance below for an example. Requirements To have attributes named in __slots__ to actually be stored in slots instead of a __dict__, a class must inherit from object (automatic in Python 3, but must be explicit in Python 2). To prevent the creation of a __dict__, you must inherit from object and all classes in the inheritance must declare __slots__ and none of them can have a '__dict__' entry. There are a lot of details if you wish to keep reading. Why use __slots__ Faster attribute access The creator of Python, Guido van Rossum, states that he actually created __slots__ for faster attribute access. It's trivial to demonstrate measurably significant speedup: import timeit class Foo(object): __slots__ = 'foo', class Bar(object): pass slotted = Foo() not_slotted = Bar() def get_set_delete_fn(obj): def get_set_delete(): obj.foo = 'foo' obj.foo del obj.foo return get_set_delete and >>> min(timeit.repeat(get_set_delete_fn(slotted))) 0.2846834529991611 >>> min(timeit.repeat(get_set_delete_fn(not_slotted))) 0.3664822799983085 The slotted access is almost 30% faster in Python 3.5 on Ubuntu. >>> 0.3664822799983085 / 0.2846834529991611 1.2873325658284342 In Python 2 on Windows I have measured it about 15% faster. Memory Savings Another purpose of __slots__ is to reduce the space in memory that each object instance takes up. My own contribution to the documentation clearly states the reasons behind this: The space saved over using __dict__ can be significant. SQLAlchemy attributes a lot of memory savings to __slots__. To verify this, using the Anaconda distribution of Python 2.7 on Ubuntu Linux, with guppy.hpy (aka heapy) and sys.getsizeof, the size of a class instance without __slots__ declared, and nothing else, is 64 bytes. That does not include the __dict__. Thank you Python for lazy evaluation again, the __dict__ is apparently not called into existence until it is referenced, but classes without data are usually useless. When called into existence, the __dict__ attribute is a minimum of 280 bytes additionally. In contrast, a class instance with __slots__ declared to be () (no data) is only 16 bytes, and 56 total bytes with one item in slots, 64 with two. For 64 bit Python, I illustrate the memory consumption in bytes in Python 2.7 and 3.6, for __slots__ and __dict__ (no slots defined) for each point where the dict grows in 3.6 (except for 0, 1, and 2 attributes): Python 2.7 Python 3.6 attrs __slots__ __dict__* __slots__ __dict__* | *(no slots defined) none 16 56 + 272† 16 56 + 112† | †if __dict__ referenced one 48 56 + 272 48 56 + 112 two 56 56 + 272 56 56 + 112 six 88 56 + 1040 88 56 + 152 11 128 56 + 1040 128 56 + 240 22 216 56 + 3344 216 56 + 408 43 384 56 + 3344 384 56 + 752 So, in spite of smaller dicts in Python 3, we see how nicely __slots__ scales for instances to save us memory, and that is a major reason you would want to use __slots__. Just for completeness of my notes, note that there is a one-time cost per slot in the class's namespace of 64 bytes in Python 2, and 72 bytes in Python 3, because slots use data descriptors like properties, called "members". >>> Foo.foo <member 'foo' of 'Foo' objects> >>> type(Foo.foo) <class 'member_descriptor'> >>> getsizeof(Foo.foo) 72 Demonstration To deny the creation of a __dict__, you must subclass object. Everything subclasses object in Python 3, but in Python 2 you had to be explicit: class Base(object): __slots__ = () now: >>> b = Base() >>> b.a = 'a' Traceback (most recent call last): File "<pyshell#38>", line 1, in <module> b.a = 'a' AttributeError: 'Base' object has no attribute 'a' Or subclass another class that defines __slots__ class Child(Base): __slots__ = ('a',) and now: c = Child() c.a = 'a' but: >>> c.b = 'b' Traceback (most recent call last): File "<pyshell#42>", line 1, in <module> c.b = 'b' AttributeError: 'Child' object has no attribute 'b' To allow __dict__ creation while subclassing slotted objects, just add '__dict__' to the __slots__ (note that slots are ordered, and you shouldn't repeat slots that are already in parent classes): class SlottedWithDict(Child): __slots__ = ('__dict__', 'b') swd = SlottedWithDict() swd.a = 'a' swd.b = 'b' swd.c = 'c' and >>> swd.__dict__ {'c': 'c'} Or you don't even need to declare __slots__ in your subclass, and you will still use slots from the parents, but not restrict the creation of a __dict__: class NoSlots(Child): pass ns = NoSlots() ns.a = 'a' ns.b = 'b' and: >>> ns.__dict__ {'b': 'b'} However, __slots__ may cause problems for multiple inheritance: class BaseA(object): __slots__ = ('a',) class BaseB(object): __slots__ = ('b',) ```python Because creating a child class from parents with both non-empty slots fails: ```python >>> class Child(BaseA, BaseB): __slots__ = () Traceback (most recent call last): File "<pyshell#68>", line 1, in <module> class Child(BaseA, BaseB): __slots__ = () TypeError: Error when calling the metaclass bases multiple bases have instance lay-out conflict If you run into this problem, You could just remove __slots__ from the parents, or if you have control of the parents, give them empty slots, or refactor to abstractions: from abc import ABC class AbstractA(ABC): __slots__ = () class BaseA(AbstractA): __slots__ = ('a',) class AbstractB(ABC): __slots__ = () class BaseB(AbstractB): __slots__ = ('b',) class Child(AbstractA, AbstractB): __slots__ = ('a', 'b') c = Child() # no problem! Add '__dict__' to __slots__ to get dynamic assignment class Foo(object): __slots__ = 'bar', 'baz', '__dict__' and now: >>> foo = Foo() >>> foo.boink = 'boink' So with '__dict__' in slots we lose some of the size benefits with the upside of having dynamic assignment and still having slots for the names we do expect. When you inherit from an object that isn't slotted, you get the same sort of semantics when you use __slots__ - names that are in __slots__ point to slotted values, while any other values are put in the instance's __dict__. Avoiding __slots__ because you want to be able to add attributes on the fly is actually not a good reason - just add "__dict__" to your __slots__ if this is required. You can similarly add __weakref__ to __slots__ explicitly if you need that feature. Set to empty tuple when subclassing a namedtuple The namedtuple builtin make immutable instances that are very lightweight (essentially, the size of tuples) but to get the benefits, you need to do it yourself if you subclass them: from collections import namedtuple class MyNT(namedtuple('MyNT', 'bar baz')): """MyNT is an immutable and lightweight object""" __slots__ = () usage: >>> nt = MyNT('bar', 'baz') >>> nt.bar 'bar' >>> nt.baz 'baz' And trying to assign an unexpected attribute raises an AttributeError because we have prevented the creation of __dict__: >>> nt.quux = 'quux' Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'MyNT' object has no attribute 'quux' You can allow __dict__ creation by leaving off __slots__ = (), but you can't use non-empty __slots__ with subtypes of tuple. Biggest Caveat: Multiple inheritance Even when non-empty slots are the same for multiple parents, they cannot be used together: class Foo(object): __slots__ = 'foo', 'bar' class Bar(object): __slots__ = 'foo', 'bar' # alas, would work if empty, i.e. () >>> class Baz(Foo, Bar): pass Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: Error when calling the metaclass bases multiple bases have instance lay-out conflict Using an empty __slots__ in the parent seems to provide the most flexibility, allowing the child to choose to prevent or allow (by adding '__dict__' to get dynamic assignment, see section above) the creation of a __dict__: class Foo(object): __slots__ = () class Bar(object): __slots__ = () class Baz(Foo, Bar): __slots__ = ('foo', 'bar') b = Baz() b.foo, b.bar = 'foo', 'bar' You don't have to have slots - so if you add them, and remove them later, it shouldn't cause any problems. Going out on a limb here: If you're composing mixins or using abstract base classes, which aren't intended to be instantiated, an empty __slots__ in those parents seems to be the best way to go in terms of flexibility for subclassers. To demonstrate, first, let's create a class with code we'd like to use under multiple inheritance class AbstractBase: __slots__ = () def __init__(self, a, b): self.a = a self.b = b def __repr__(self): return f'{type(self).__name__}({repr(self.a)}, {repr(self.b)})' We could use the above directly by inheriting and declaring the expected slots: class Foo(AbstractBase): __slots__ = 'a', 'b' But we don't care about that, that's trivial single inheritance, we need another class we might also inherit from, maybe with a noisy attribute: class AbstractBaseC: __slots__ = () @property def c(self): print('getting c!') return self._c @c.setter def c(self, arg): print('setting c!') self._c = arg Now if both bases had nonempty slots, we couldn't do the below. (In fact, if we wanted, we could have given AbstractBase nonempty slots a and b, and left them out of the below declaration - leaving them in would be wrong): class Concretion(AbstractBase, AbstractBaseC): __slots__ = 'a b _c'.split() And now we have functionality from both via multiple inheritance, and can still deny __dict__ and __weakref__ instantiation: >>> c = Concretion('a', 'b') >>> c.c = c setting c! >>> c.c getting c! Concretion('a', 'b') >>> c.d = 'd' Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'Concretion' object has no attribute 'd' Other cases to avoid slots Avoid them when you want to perform __class__ assignment with another class that doesn't have them (and you can't add them) unless the slot layouts are identical. (I am very interested in learning who is doing this and why.) Avoid them if you want to subclass variable length builtins like long, tuple, or str, and you want to add attributes to them. Avoid them if you insist on providing default values via class attributes for instance variables. You may be able to tease out further caveats from the rest of the __slots__ documentation. Critiques of other answers The current top answers cite outdated information and are quite hand-wavy and miss the mark in some important ways. Do not "only use __slots__ when instantiating lots of objects" I quote: "You would want to use __slots__ if you are going to instantiate a lot (hundreds, thousands) of objects of the same class." Abstract Base Classes, for example, from the collections module, are not instantiated, yet __slots__ are declared for them. Why? If a user wishes to deny __dict__ or __weakref__ creation, those things must not be available in the parent classes. __slots__ contributes to reusability when creating interfaces or mixins. It is true that many Python users aren't writing for reusability, but when you are, having the option to deny unnecessary space usage is valuable. __slots__ doesn't break pickling When pickling a slotted object, you may find it complains with a misleading TypeError: >>> pickle.loads(pickle.dumps(f)) TypeError: a class that defines __slots__ without defining __getstate__ cannot be pickled This is actually incorrect. This message comes from the oldest protocol, which is the default. You can select the latest protocol with the -1 argument. In Python 2.7 this would be 2 (which was introduced in 2.3), and in 3.6 it is 4. >>> pickle.loads(pickle.dumps(f, -1)) <__main__.Foo object at 0x1129C770> in Python 2.7: >>> pickle.loads(pickle.dumps(f, 2)) <__main__.Foo object at 0x1129C770> in Python 3.6 >>> pickle.loads(pickle.dumps(f, 4)) <__main__.Foo object at 0x1129C770> So I would keep this in mind, as it is a solved problem. Critique of the (until Oct 2, 2016) accepted answer The first paragraph is half short explanation, half predictive. Here's the only part that actually answers the question The proper use of __slots__ is to save space in objects. Instead of having a dynamic dict that allows adding attributes to objects at anytime, there is a static structure which does not allow additions after creation. This saves the overhead of one dict for every object that uses slots The second half is wishful thinking, and off the mark: While this is sometimes a useful optimization, it would be completely unnecessary if the Python interpreter was dynamic enough so that it would only require the dict when there actually were additions to the object. Python actually does something similar to this, only creating the __dict__ when it is accessed, but creating lots of objects with no data is fairly ridiculous. The second paragraph oversimplifies and misses actual reasons to avoid __slots__. The below is not a real reason to avoid slots (for actual reasons, see the rest of my answer above.): They change the behavior of the objects that have slots in a way that can be abused by control freaks and static typing weenies. It then goes on to discuss other ways of accomplishing that perverse goal with Python, not discussing anything to do with __slots__. The third paragraph is more wishful thinking. Together it is mostly off-the-mark content that the answerer didn't even author and contributes to ammunition for critics of the site. Memory usage evidence Create some normal objects and slotted objects: >>> class Foo(object): pass >>> class Bar(object): __slots__ = () Instantiate a million of them: >>> foos = [Foo() for f in xrange(1000000)] >>> bars = [Bar() for b in xrange(1000000)] Inspect with guppy.hpy().heap(): >>> guppy.hpy().heap() Partition of a set of 2028259 objects. Total size = 99763360 bytes. Index Count % Size % Cumulative % Kind (class / dict of class) 0 1000000 49 64000000 64 64000000 64 __main__.Foo 1 169 0 16281480 16 80281480 80 list 2 1000000 49 16000000 16 96281480 97 __main__.Bar 3 12284 1 987472 1 97268952 97 str ... Access the regular objects and their __dict__ and inspect again: >>> for f in foos: ... f.__dict__ >>> guppy.hpy().heap() Partition of a set of 3028258 objects. Total size = 379763480 bytes. Index Count % Size % Cumulative % Kind (class / dict of class) 0 1000000 33 280000000 74 280000000 74 dict of __main__.Foo 1 1000000 33 64000000 17 344000000 91 __main__.Foo 2 169 0 16281480 4 360281480 95 list 3 1000000 33 16000000 4 376281480 99 __main__.Bar 4 12284 0 987472 0 377268952 99 str ... This is consistent with the history of Python, from Unifying types and classes in Python 2.2. If you subclass a built-in type, extra space is automatically added to the instances to accomodate __dict__ and __weakrefs__. (The __dict__ is not initialized until you use it though, so you shouldn't worry about the space occupied by an empty dictionary for each instance you create.) If you don't need this extra space, you can add the phrase "__slots__ = []" to your class.


Instruction:
How do I sort a list of objects based on an attribute of the objects?

I have a list of Python objects that I want to sort by a specific attribute of each object: [Tag(name="toe", count=10), Tag(name="leg", count=2), ...] How do I sort the list by .count in descending order?

Output: To sort the list in place: orig_list.sort(key=lambda x: x.count, reverse=True) To return a new list, use sorted: new_list = sorted(orig_list, key=lambda x: x.count, reverse=True) Explanation: key=lambda x: x.count sorts by count. reverse=True sorts in descending order. More on sorting by keys.


Instruction:
Remove all whitespace in a string

I want to eliminate all the whitespace from a string, on both ends, and in between words. I have this Python code: def my_handle(self): sentence = ' hello apple ' sentence.strip() But that only eliminates the whitespace on both sides of the string. How do I remove all whitespace?

Output: If you want to remove leading and ending whitespace, use str.strip(): >>> " hello apple ".strip() 'hello apple' If you want to remove all space characters, use str.replace() (NB this only removes the “normal” ASCII space character ' ' U+0020 but not any other whitespace): >>> " hello apple ".replace(" ", "") 'helloapple' If you want to remove all whitespace and then leave a single space character between words, use str.split() followed by str.join(): >>> " ".join(" hello apple ".split()) 'hello apple' If you want to remove all whitespace then change the above leading " " to "": >>> "".join(" hello apple ".split()) 'helloapple'


Instruction:
How do you test that a Python function throws an exception?

How does one write a unit test that fails only if a function doesn't throw an expected exception?

Output: Use TestCase.assertRaises from the unittest module, for example: import mymod class MyTestCase(unittest.TestCase): def test1(self): self.assertRaises(SomeCoolException, mymod.myfunc)


Instruction:
Get difference between two lists with Unique Entries

I have two lists in Python: temp1 = ['One', 'Two', 'Three', 'Four'] temp2 = ['One', 'Two'] Assuming the elements in each list are unique, I want to create a third list with items from the first list which are not in the second list: temp3 = ['Three', 'Four'] Are there any fast ways without cycles and checking?

Output: To get elements which are in temp1 but not in temp2 (assuming uniqueness of the elements in each list): In [5]: list(set(temp1) - set(temp2)) Out[5]: ['Four', 'Three'] Beware that it is asymmetric : In [5]: set([1, 2]) - set([2, 3]) Out[5]: set([1]) where you might expect/want it to equal set([1, 3]). If you do want set([1, 3]) as your answer, you can use set([1, 2]).symmetric_difference(set([2, 3])).


Instruction:
How to concatenate (join) items in a list to a single string

How do I concatenate a list of strings into a single string? For example, given ['this', 'is', 'a', 'sentence'], how do I get "this-is-a-sentence"? For handling a few strings in separate variables, see How do I append one string to another in Python?. For the opposite process - creating a list from a string - see How do I split a string into a list of characters? or How do I split a string into a list of words? as appropriate.

Output: Use str.join: >>> words = ['this', 'is', 'a', 'sentence'] >>> '-'.join(words) 'this-is-a-sentence' >>> ' '.join(words) 'this is a sentence'


Instruction:
What is a cross-platform way to get the home directory?

I need to get the location of the home directory of the current logged-on user. Currently, I've been using the following on Linux: os.getenv("HOME") However, this does not work on Windows. What is the correct cross-platform way to do this ?

Output: On Python 3.5+ you can use pathlib.Path.home(): from pathlib import Path home = Path.home() # example usage: with open(home / ".ssh" / "known_hosts") as f: lines = f.readlines() to get a pathlib.PosixPath object. Use str() to convert to a string if necessary. On older Python versions, you can use os.path.expanduser. from os.path import expanduser home = expanduser("~")


Instruction:
How do I trim whitespace?

Is there a Python function that will trim whitespace (spaces and tabs) from a string? So that given input " \t example string\t " becomes "example string".

Output: For whitespace on both sides, use str.strip: s = " \t a string example\t " s = s.strip() For whitespace on the right side, use str.rstrip: s = s.rstrip() For whitespace on the left side, use str.lstrip: s = s.lstrip() You can provide an argument to strip arbitrary characters to any of these functions, like this: s = s.strip(' \t\n\r') This will strip any space, \t, \n, or \r characters from both sides of the string. The examples above only remove strings from the left-hand and right-hand sides of strings. If you want to also remove characters from the middle of a string, try re.sub: import re print(re.sub('[\s+]', '', s)) That should print out: astringexample


Instruction:
Is there a simple way to delete a list element by value?

I want to remove a value from a list if it exists in the list (which it may not). a = [1, 2, 3, 4] b = a.index(6) del a[b] print(a) The above gives the error: ValueError: list.index(x): x not in list So I have to do this: a = [1, 2, 3, 4] try: b = a.index(6) del a[b] except: pass print(a) But is there not a simpler way to do this?

Output: To remove the first occurrence of an element, use list.remove: >>> xs = ['a', 'b', 'c', 'd'] >>> xs.remove('b') >>> print(xs) ['a', 'c', 'd'] To remove all occurrences of an element, use a list comprehension: >>> xs = ['a', 'b', 'c', 'd', 'b', 'b', 'b', 'b'] >>> xs = [x for x in xs if x != 'b'] >>> print(xs) ['a', 'c', 'd']


Instruction:
Convert list of dictionaries to a pandas DataFrame

How can I convert a list of dictionaries into a DataFrame? I want to turn [{'points': 50, 'time': '5:00', 'year': 2010}, {'points': 25, 'time': '6:00', 'month': "february"}, {'points':90, 'time': '9:00', 'month': 'january'}, {'points_h1':20, 'month': 'june'}] into month points points_h1 time year 0 NaN 50 NaN 5:00 2010 1 february 25 NaN 6:00 NaN 2 january 90 NaN 9:00 NaN 3 june NaN 20 NaN NaN

Output: If ds is a list of dicts: df = pd.DataFrame(ds) Note: this does not work with nested data.


Instruction:
Writing a pandas DataFrame to CSV file

I have a dataframe in pandas which I would like to write to a CSV file. I am doing this using: df.to_csv('out.csv') And getting the following error: UnicodeEncodeError: 'ascii' codec can't encode character u'\u03b1' in position 20: ordinal not in range(128) Is there any way to get around this easily (i.e. I have unicode characters in my data frame)? And is there a way to write to a tab delimited file instead of a CSV using e.g. a 'to-tab' method (that I don't think exists)?

Output: To delimit by a tab you can use the sep argument of to_csv: df.to_csv(file_name, sep='\t') To use a specific encoding (e.g. 'utf-8') use the encoding argument: df.to_csv(file_name, sep='\t', encoding='utf-8') In many cases you will want to remove the index and add a header: df.to_csv(file_name, sep='\t', encoding='utf-8', index=False, header=True)


Instruction:
What's the difference between lists and tuples?

What are the differences between lists and tuples, and what are their respective advantages and disadvantages?

Output: Apart from tuples being immutable there is also a semantic distinction that should guide their usage. Tuples are heterogeneous data structures (i.e., their entries have different meanings), while lists are homogeneous sequences. Tuples have structure, lists have order. Using this distinction makes code more explicit and understandable. One example would be pairs of page and line number to reference locations in a book, e.g.: my_location = (42, 11) # page number, line number You can then use this as a key in a dictionary to store notes on locations. A list on the other hand could be used to store multiple locations. Naturally one might want to add or remove locations from the list, so it makes sense that lists are mutable. On the other hand it doesn't make sense to add or remove items from an existing location - hence tuples are immutable. There might be situations where you want to change items within an existing location tuple, for example when iterating through the lines of a page. But tuple immutability forces you to create a new location tuple for each new value. This seems inconvenient on the face of it, but using immutable data like this is a cornerstone of value types and functional programming techniques, which can have substantial advantages. There are some interesting articles on this issue, e.g. "Python Tuples are Not Just Constant Lists" or "Understanding tuples vs. lists in Python". The official Python documentation also mentions this "Tuples are immutable, and usually contain an heterogeneous sequence ...". In a statically typed language like Haskell the values in a tuple generally have different types and the length of the tuple must be fixed. In a list the values all have the same type and the length is not fixed. So the difference is very obvious. Finally there is the namedtuple in Python, which makes sense because a tuple is already supposed to have structure. This underlines the idea that tuples are a light-weight alternative to classes and instances.


Instruction:
How can I parse a YAML file in Python

How can I parse a YAML file in Python?

Output: The easiest method without relying on C headers is PyYaml (documentation), which can be installed via pip install pyyaml: import yaml with open("example.yaml") as stream: try: print(yaml.safe_load(stream)) except yaml.YAMLError as exc: print(exc) yaml.load() also exists, but yaml.safe_load() should always be preferred to avoid introducing the possibility for arbitrary code execution. So unless you explicitly need the arbitrary object serialization/deserialization use safe_load. The PyYaml project supports versions up through the YAML 1.1 specification. If YAML 1.2 specification support is needed, see ruamel.yaml as noted in this answer. Also, you could also use a drop in replacement for pyyaml, that keeps your yaml file ordered the same way you had it, called oyaml. View snyk of oyaml here


Instruction:
UnicodeDecodeError: 'charmap' codec can't decode byte X in position Y: character maps to <undefined>

I'm trying to get a Python 3 program to do some manipulations with a text file filled with information. However, when trying to read the file I get the following error: Traceback (most recent call last): File "SCRIPT LOCATION", line NUMBER, in <module> text = file.read() File "C:\Python31\lib\encodings\cp1252.py", line 23, in decode return codecs.charmap_decode(input,self.errors,decoding_table)[0] UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 2907500: character maps to `<undefined>` Editor's note: After reading this Q&A, see How to determine the encoding of text if you need help figuring out the encoding of the file you are trying to open.

Output: The file in question is not using the CP1252 encoding. It's using another encoding. Which one you have to figure out yourself. Common ones are Latin-1 and UTF-8. Since 0x90 doesn't actually mean anything in Latin-1, UTF-8 (where 0x90 is a continuation byte) is more likely. You specify the encoding when you open the file: file = open(filename, encoding="utf-8")


Instruction:
How do I check the versions of Python modules?

I installed the Python modules construct and statlib using setuptools: sudo apt-get install python-setuptools sudo easy_install statlib sudo easy_install construct How do I check their versions from the command line?

Output: Use pip instead of easy_install. With pip, list all installed packages and their versions via: pip freeze On most Linux systems, you can pipe this to grep (or findstr on Windows) to find the row for the particular package you're interested in. Linux: pip freeze | grep lxml lxml==2.3 Windows: pip freeze | findstr lxml lxml==2.3 For an individual module, you can try the __version__ attribute. However, there are modules without it: python -c "import requests; print(requests.__version__)" 2.14.2 python -c "import lxml; print(lxml.__version__)" Traceback (most recent call last): File "<string>", line 1, in <module> AttributeError: 'module' object has no attribute '__version__' Lastly, as the commands in your question are prefixed with sudo, it appears you're installing to the global python environment. I strongly advise to take look into Python virtual environment managers, for example virtualenvwrapper.


Instruction:
How can I POST JSON data with Python's Requests library?

I need to POST JSON content from a client to a server. I'm using Python 2.7.1 and simplejson. The client is using Requests. The server is CherryPy. I can GET hard-coded JSON content from the server (the code is not shown), but when I try to POST JSON content to the server, I get "400 Bad Request". Here is my client code: data = {'sender': 'Alice', 'receiver': 'Bob', 'message': 'We did it!'} data_json = simplejson.dumps(data) payload = {'json_payload': data_json} r = requests.post("http://localhost:8080", data=payload) Here is the server code. class Root(object): def __init__(self, content): self.content = content print self.content # This works exposed = True def GET(self): cherrypy.response.headers['Content-Type'] = 'application/json' return simplejson.dumps(self.content) def POST(self): self.content = simplejson.loads(cherrypy.request.body.read()) How can I fix this problem?

Output: Starting with Requests version 2.4.2, you can use the json= parameter (which takes a dictionary) instead of data= (which takes a string) in the call: >>> import requests >>> r = requests.post('http://httpbin.org/post', json={"key": "value"}) >>> r.status_code 200 >>> r.json() {'args': {}, 'data': '{"key": "value"}', 'files': {}, 'form': {}, 'headers': {'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'Connection': 'close', 'Content-Length': '16', 'Content-Type': 'application/json', 'Host': 'httpbin.org', 'User-Agent': 'python-requests/2.4.3 CPython/3.4.0', 'X-Request-Id': 'xx-xx-xx'}, 'json': {'key': 'value'}, 'origin': 'x.x.x.x', 'url': 'http://httpbin.org/post'}


Instruction:
Getting today's date in YYYY-MM-DD in Python?

Is there a nicer way than the following to return today's date in the YYYY-MM-DD format? str(datetime.datetime.today()).split()[0]

Output: Use strftime: >>> from datetime import datetime >>> datetime.today().strftime('%Y-%m-%d') '2021-01-26' To also include a zero-padded Hour:Minute:Second at the end: >>> datetime.today().strftime('%Y-%m-%d %H:%M:%S') '2021-01-26 16:50:03' To get the UTC date and time: >>> datetime.utcnow().strftime('%Y-%m-%d %H:%M:%S') '2021-01-27 00:50:03'


Instruction:
What does functools.wraps do?

In a comment on this answer to another question, someone said that they weren't sure what functools.wraps was doing. So, I'm asking this question so that there will be a record of it on StackOverflow for future reference: what does functools.wraps do, exactly?

Output: When you use a decorator, you're replacing one function with another. In other words, if you have a decorator def logged(func): def with_logging(*args, **kwargs): print(func.__name__ + " was called") return func(*args, **kwargs) return with_logging then when you say @logged def f(x): """does some math""" return x + x * x it's exactly the same as saying def f(x): """does some math""" return x + x * x f = logged(f) and your function f is replaced with the function with_logging. Unfortunately, this means that if you then say print(f.__name__) it will print with_logging because that's the name of your new function. In fact, if you look at the docstring for f, it will be blank because with_logging has no docstring, and so the docstring you wrote won't be there anymore. Also, if you look at the pydoc result for that function, it won't be listed as taking one argument x; instead it'll be listed as taking *args and **kwargs because that's what with_logging takes. If using a decorator always meant losing this information about a function, it would be a serious problem. That's why we have functools.wraps. This takes a function used in a decorator and adds the functionality of copying over the function name, docstring, arguments list, etc. And since wraps is itself a decorator, the following code does the correct thing: from functools import wraps def logged(func): @wraps(func) def with_logging(*args, **kwargs): print(func.__name__ + " was called") return func(*args, **kwargs) return with_logging @logged def f(x): """does some math""" return x + x * x print(f.__name__) # prints 'f' print(f.__doc__) # prints 'does some math'


Instruction:
How to use glob() to find files recursively?

I would like to list all files recursively in a directory. I currently have a directory structure like this: src/main.c src/dir/file1.c src/another-dir/file2.c src/another-dir/nested/files/file3.c I've tried to do the following: from glob import glob glob(os.path.join('src','*.c')) But this will only get be files directly in the src subfolder, e.g. I get main.c but I will not get file1.c, file2.c etc. from glob import glob glob(os.path.join('src','*.c')) glob(os.path.join('src','*','*.c')) glob(os.path.join('src','*','*','*.c')) glob(os.path.join('src','*','*','*','*.c')) But this is obviously limited and clunky, how can I do this properly?

Output: There are a couple of ways: pathlib.Path().rglob() Use pathlib.Path().rglob() from the pathlib module, which was introduced in Python 3.5. from pathlib import Path for path in Path('src').rglob('*.c'): print(path.name) glob.glob() If you don't want to use pathlib, use glob.glob(): from glob import glob for filename in glob('src/**/*.c', recursive=True): print(filename) For cases where matching files beginning with a dot (.); like files in the current directory or hidden files on Unix based system, use the os.walk() solution below. os.walk() For older Python versions, use os.walk() to recursively walk a directory and fnmatch.filter() to match against a simple expression: import fnmatch import os matches = [] for root, dirnames, filenames in os.walk('src'): for filename in fnmatch.filter(filenames, '*.c'): matches.append(os.path.join(root, filename)) This version should also be faster depending on how many files you have, as the pathlib module has a bit of overhead over os.walk().


Instruction:
Convert date to datetime in Python

Is there a built-in method for converting a date to a datetime in Python, for example getting the datetime for the midnight of the given date? The opposite conversion is easy: datetime has a .date() method. Do I really have to manually call datetime(d.year, d.month, d.day)?

Output: You can use datetime.combine(date, time); for the time, you create a datetime.time object initialized to midnight. from datetime import date from datetime import datetime dt = datetime.combine(date.today(), datetime.min.time())


Instruction:
How do I expand the output display to see more columns of a Pandas DataFrame?

Is there a way to widen the display of output in either interactive or script-execution mode? Specifically, I am using the describe() function on a Pandas DataFrame. When the DataFrame is five columns (labels) wide, I get the descriptive statistics that I want. However, if the DataFrame has any more columns, the statistics are suppressed and something like this is returned: >> Index: 8 entries, count to max >> Data columns: >> x1 8 non-null values >> x2 8 non-null values >> x3 8 non-null values >> x4 8 non-null values >> x5 8 non-null values >> x6 8 non-null values >> x7 8 non-null values The "8" value is given whether there are 6 or 7 columns. What does the "8" refer to? I have already tried dragging the IDLE window larger, as well as increasing the "Configure IDLE" width options, to no avail.

Output: (For Pandas versions before 0.23.4, see at bottom.) Use pandas.set_option(optname, val), or equivalently pd.options.<opt.hierarchical.name> = val. Like: import pandas as pd pd.set_option('display.max_rows', 500) pd.set_option('display.max_columns', 500) pd.set_option('display.width', 1000) Pandas will try to autodetect the size of your terminal window if you set pd.options.display.width = 0. Here is the help for set_option: set_option(pat,value) - Sets the value of the specified option Available options: display.[chop_threshold, colheader_justify, column_space, date_dayfirst, date_yearfirst, encoding, expand_frame_repr, float_format, height, line_width, max_columns, max_colwidth, max_info_columns, max_info_rows, max_rows, max_seq_items, mpl_style, multi_sparse, notebook_repr_html, pprint_nest_depth, precision, width] mode.[sim_interactive, use_inf_as_null] Parameters ---------- pat - str/regexp which should match a single option. Note: partial matches are supported for convenience, but unless you use the full option name (e.g., *x.y.z.option_name*), your code may break in future versions if new options with similar names are introduced. value - new value of option. Returns ------- None Raises ------ KeyError if no such option exists display.chop_threshold: [default: None] [currently: None] : float or None if set to a float value, all float values smaller then the given threshold will be displayed as exactly 0 by repr and friends. display.colheader_justify: [default: right] [currently: right] : 'left'/'right' Controls the justification of column headers. used by DataFrameFormatter. display.column_space: [default: 12] [currently: 12]No description available. display.date_dayfirst: [default: False] [currently: False] : boolean When True, prints and parses dates with the day first, eg 20/01/2005 display.date_yearfirst: [default: False] [currently: False] : boolean When True, prints and parses dates with the year first, e.g., 2005/01/20 display.encoding: [default: UTF-8] [currently: UTF-8] : str/unicode Defaults to the detected encoding of the console. Specifies the encoding to be used for strings returned by to_string, these are generally strings meant to be displayed on the console. display.expand_frame_repr: [default: True] [currently: True] : boolean Whether to print out the full DataFrame repr for wide DataFrames across multiple lines, `max_columns` is still respected, but the output will wrap-around across multiple "pages" if it's width exceeds `display.width`. display.float_format: [default: None] [currently: None] : callable The callable should accept a floating point number and return a string with the desired format of the number. This is used in some places like SeriesFormatter. See core.format.EngFormatter for an example. display.height: [default: 60] [currently: 1000] : int Deprecated. (Deprecated, use `display.height` instead.) display.line_width: [default: 80] [currently: 1000] : int Deprecated. (Deprecated, use `display.width` instead.) display.max_columns: [default: 20] [currently: 500] : int max_rows and max_columns are used in __repr__() methods to decide if to_string() or info() is used to render an object to a string. In case python/IPython is running in a terminal this can be set to 0 and Pandas will correctly auto-detect the width the terminal and swap to a smaller format in case all columns would not fit vertically. The IPython notebook, IPython qtconsole, or IDLE do not run in a terminal and hence it is not possible to do correct auto-detection. 'None' value means unlimited. display.max_colwidth: [default: 50] [currently: 50] : int The maximum width in characters of a column in the repr of a Pandas data structure. When the column overflows, a "..." placeholder is embedded in the output. display.max_info_columns: [default: 100] [currently: 100] : int max_info_columns is used in DataFrame.info method to decide if per column information will be printed. display.max_info_rows: [default: 1690785] [currently: 1690785] : int or None max_info_rows is the maximum number of rows for which a frame will perform a null check on its columns when repr'ing To a console. The default is 1,000,000 rows. So, if a DataFrame has more 1,000,000 rows there will be no null check performed on the columns and thus the representation will take much less time to display in an interactive session. A value of None means always perform a null check when repr'ing. display.max_rows: [default: 60] [currently: 500] : int This sets the maximum number of rows Pandas should output when printing out various output. For example, this value determines whether the repr() for a dataframe prints out fully or just a summary repr. 'None' value means unlimited. display.max_seq_items: [default: None] [currently: None] : int or None when pretty-printing a long sequence, no more then `max_seq_items` will be printed. If items are ommitted, they will be denoted by the addition of "..." to the resulting string. If set to None, the number of items to be printed is unlimited. display.mpl_style: [default: None] [currently: None] : bool Setting this to 'default' will modify the rcParams used by matplotlib to give plots a more pleasing visual style by default. Setting this to None/False restores the values to their initial value. display.multi_sparse: [default: True] [currently: True] : boolean "sparsify" MultiIndex display (don't display repeated elements in outer levels within groups) display.notebook_repr_html: [default: True] [currently: True] : boolean When True, IPython notebook will use html representation for Pandas objects (if it is available). display.pprint_nest_depth: [default: 3] [currently: 3] : int Controls the number of nested levels to process when pretty-printing display.precision: [default: 7] [currently: 7] : int Floating point output precision (number of significant digits). This is only a suggestion display.width: [default: 80] [currently: 1000] : int Width of the display in characters. In case python/IPython is running in a terminal this can be set to None and Pandas will correctly auto-detect the width. Note that the IPython notebook, IPython qtconsole, or IDLE do not run in a terminal and hence it is not possible to correctly detect the width. mode.sim_interactive: [default: False] [currently: False] : boolean Whether to simulate interactive mode for purposes of testing mode.use_inf_as_null: [default: False] [currently: False] : boolean True means treat None, NaN, INF, -INF as null (old way), False means None and NaN are null, but INF, -INF are not null (new way). Call def: pd.set_option(self, *args, **kwds) Older version information Much of this has been deprecated. As @bmu mentioned, Pandas auto detects (by default) the size of the display area, a summary view will be used when an object repr does not fit on the display. You mentioned resizing the IDLE window, to no effect. If you do print df.describe().to_string() does it fit on the IDLE window? The terminal size is determined by pandas.util.terminal.get_terminal_size() (deprecated and removed), this returns a tuple containing the (width, height) of the display. Does the output match the size of your IDLE window? There might be an issue (there was one before when running a terminal in Emacs). Note that it is possible to bypass the autodetect, pandas.set_printoptions(max_rows=200, max_columns=10) will never switch to summary view if number of rows, columns does not exceed the given limits. The max_colwidth option helps in seeing untruncated form of each column.


Instruction:
Find all files in a directory with extension .txt in Python

How can I find all the files in a directory having the extension .txt in python?

Output: You can use glob: import glob, os os.chdir("/mydir") for file in glob.glob("*.txt"): print(file) or simply os.listdir: import os for file in os.listdir("/mydir"): if file.endswith(".txt"): print(os.path.join("/mydir", file)) or if you want to traverse directory, use os.walk: import os for root, dirs, files in os.walk("/mydir"): for file in files: if file.endswith(".txt"): print(os.path.join(root, file))


Instruction:
Running unittest with typical test directory structure

The very common directory structure for even a simple Python module seems to be to separate the unit tests into their own test directory: new_project/ antigravity/ antigravity.py test/ test_antigravity.py setup.py etc. My question is simply What's the usual way of actually running the tests? I suspect this is obvious to everyone except me, but you can't just run python test_antigravity.py from the test directory as its import antigravity will fail as the module is not on the path. I know I could modify PYTHONPATH and other search path related tricks, but I can't believe that's the simplest way - it's fine if you're the developer but not realistic to expect your users to use if they just want to check the tests are passing. The other alternative is just to copy the test file into the other directory, but it seems a bit dumb and misses the point of having them in a separate directory to start with. So, if you had just downloaded the source to my new project how would you run the unit tests? I'd prefer an answer that would let me say to my users: "To run the unit tests do X."

Output: The best solution in my opinion is to use the unittest command line interface which will add the directory to the sys.path so you don't have to (done in the TestLoader class). For example for a directory structure like this: new_project ├── antigravity.py └── test_antigravity.py You can just run: $ cd new_project $ python -m unittest test_antigravity For a directory structure like yours: new_project ├── antigravity │ ├── __init__.py # make it a package │ └── antigravity.py └── test ├── __init__.py # also make test a package └── test_antigravity.py And in the test modules inside the test package, you can import the antigravity package and its modules as usual: # import the package import antigravity # import the antigravity module from antigravity import antigravity # or an object inside the antigravity module from antigravity.antigravity import my_object Running a single test module: To run a single test module, in this case test_antigravity.py: $ cd new_project $ python -m unittest test.test_antigravity Just reference the test module the same way you import it. Running a single test case or test method: Also you can run a single TestCase or a single test method: $ python -m unittest test.test_antigravity.GravityTestCase $ python -m unittest test.test_antigravity.GravityTestCase.test_method Running all tests: You can also use test discovery which will discover and run all the tests for you, they must be modules or packages named test*.py (can be changed with the -p, --pattern flag): $ cd new_project $ python -m unittest discover $ # Also works without discover for Python 3 $ # as suggested by @Burrito in the comments $ python -m unittest This will run all the test*.py modules inside the test package. Here you can find the updated official documentation of discovery.


Instruction:
How do you get the logical xor of two variables in Python?

How do you get the logical xor of two variables in Python? For example, I have two variables that I expect to be strings. I want to test that only one of them contains a True value (is not None or an empty string): str1 = raw_input("Enter string one:") str2 = raw_input("Enter string two:") if logical_xor(str1, str2): print "ok" else: print "bad" The ^ operator is bitwise, and not defined on all objects: >>> 1 ^ 1 0 >>> 2 ^ 1 3 >>> "abc" ^ "" Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: unsupported operand type(s) for ^: 'str' and 'str'

Output: If you're already normalizing the inputs to booleans, then != is xor. bool(a) != bool(b)


Instruction:
Is there any way to kill a Thread?

Is it possible to terminate a running thread without setting/checking any flags/semaphores/etc.?

Output: It is generally a bad pattern to kill a thread abruptly, in Python, and in any language. Think of the following cases: the thread is holding a critical resource that must be closed properly the thread has created several other threads that must be killed as well. The nice way of handling this, if you can afford it (if you are managing your own threads), is to have an exit_request flag that each thread checks on a regular interval to see if it is time for it to exit. For example: import threading class StoppableThread(threading.Thread): """Thread class with a stop() method. The thread itself has to check regularly for the stopped() condition.""" def __init__(self, *args, **kwargs): super(StoppableThread, self).__init__(*args, **kwargs) self._stop_event = threading.Event() def stop(self): self._stop_event.set() def stopped(self): return self._stop_event.is_set() In this code, you should call stop() on the thread when you want it to exit, and wait for the thread to exit properly using join(). The thread should check the stop flag at regular intervals. There are cases, however, when you really need to kill a thread. An example is when you are wrapping an external library that is busy for long calls, and you want to interrupt it. The following code allows (with some restrictions) to raise an Exception in a Python thread: def _async_raise(tid, exctype): '''Raises an exception in the threads with id tid''' if not inspect.isclass(exctype): raise TypeError("Only types can be raised (not instances)") res = ctypes.pythonapi.PyThreadState_SetAsyncExc(ctypes.c_long(tid), ctypes.py_object(exctype)) if res == 0: raise ValueError("invalid thread id") elif res != 1: # "if it returns a number greater than one, you're in trouble, # and you should call it again with exc=NULL to revert the effect" ctypes.pythonapi.PyThreadState_SetAsyncExc(ctypes.c_long(tid), None) raise SystemError("PyThreadState_SetAsyncExc failed") class ThreadWithExc(threading.Thread): '''A thread class that supports raising an exception in the thread from another thread. ''' def _get_my_tid(self): """determines this (self's) thread id CAREFUL: this function is executed in the context of the caller thread, to get the identity of the thread represented by this instance. """ if not self.is_alive(): # Note: self.isAlive() on older version of Python raise threading.ThreadError("the thread is not active") # do we have it cached? if hasattr(self, "_thread_id"): return self._thread_id # no, look for it in the _active dict for tid, tobj in threading._active.items(): if tobj is self: self._thread_id = tid return tid # TODO: in python 2.6, there's a simpler way to do: self.ident raise AssertionError("could not determine the thread's id") def raise_exc(self, exctype): """Raises the given exception type in the context of this thread. If the thread is busy in a system call (time.sleep(), socket.accept(), ...), the exception is simply ignored. If you are sure that your exception should terminate the thread, one way to ensure that it works is: t = ThreadWithExc( ... ) ... t.raise_exc( SomeException ) while t.isAlive(): time.sleep( 0.1 ) t.raise_exc( SomeException ) If the exception is to be caught by the thread, you need a way to check that your thread has caught it. CAREFUL: this function is executed in the context of the caller thread, to raise an exception in the context of the thread represented by this instance. """ _async_raise( self._get_my_tid(), exctype ) (Based on Killable Threads by Tomer Filiba. The quote about the return value of PyThreadState_SetAsyncExc appears to be from an old version of Python.) As noted in the documentation, this is not a magic bullet because if the thread is busy outside the Python interpreter, it will not catch the interruption. A good usage pattern of this code is to have the thread catch a specific exception and perform the cleanup. That way, you can interrupt a task and still have proper cleanup.


Instruction:
How do I use a decimal step value for range()?

How do I iterate between 0 and 1 by a step of 0.1? This says that the step argument cannot be zero: for i in range(0, 1, 0.1): print(i)

Output: Rather than using a decimal step directly, it's much safer to express this in terms of how many points you want. Otherwise, floating-point rounding error is likely to give you a wrong result. Use the linspace function from the NumPy library (which isn't part of the standard library but is relatively easy to obtain). linspace takes a number of points to return, and also lets you specify whether or not to include the right endpoint: >>> np.linspace(0,1,11) array([ 0. , 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1. ]) >>> np.linspace(0,1,10,endpoint=False) array([ 0. , 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]) If you really want to use a floating-point step value, use numpy.arange: >>> import numpy as np >>> np.arange(0.0, 1.0, 0.1) array([ 0. , 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]) Floating-point rounding error will cause problems, though. Here's a simple case where rounding error causes arange to produce a length-4 array when it should only produce 3 numbers: >>> numpy.arange(1, 1.3, 0.1) array([1. , 1.1, 1.2, 1.3])


Instruction:
Convert hex string to integer in Python

How do I convert a hex string to an integer? "0xffff" ⟶ 65535 "ffff" ⟶ 65535

Output: Without the 0x prefix, you need to specify the base explicitly, otherwise there's no way to tell: x = int("deadbeef", 16) With the 0x prefix, Python can distinguish hex and decimal automatically: >>> print(int("0xdeadbeef", 0)) 3735928559 >>> print(int("10", 0)) 10 (You must specify 0 as the base in order to invoke this prefix-guessing behavior; if you omit the second parameter, int() will assume base-10.)


Instruction:
Writing a list to a file with Python, with newlines

How do I write a list to a file? writelines() doesn't insert newline characters, so I need to do: f.writelines([f"{line}\n" for line in lines])

Output: Use a loop: with open('your_file.txt', 'w') as f: for line in lines: f.write(f"{line}\n") For Python <3.6: with open('your_file.txt', 'w') as f: for line in lines: f.write("%s\n" % line) For Python 2, one may also use: with open('your_file.txt', 'w') as f: for line in lines: print >> f, line If you're keen on a single function call, at least remove the square brackets [], so that the strings to be printed get made one at a time (a genexp rather than a listcomp) -- no reason to take up all the memory required to materialize the whole list of strings.


Instruction:
How to identify which OS Python is running on

What do I need to look at to see whether I'm on Windows or Unix, etc.?

Output: >>> import os >>> os.name 'posix' >>> import platform >>> platform.system() 'Linux' >>> platform.release() '2.6.22-15-generic' The output of platform.system() is as follows: Linux: Linux Mac: Darwin Windows: Windows See: platform — Access to underlying platform’s identifying data


Instruction:
How do I update/upgrade pip itself from inside my virtual environment?

I'm able to update pip-managed packages, but how do I update pip itself? According to pip --version, I currently have pip 1.1 installed in my virtualenv and I want to update to the latest version. What's the command for that? Do I need to use distribute or is there a native pip or virtualenv command? I've already tried pip update and pip update pip with no success.

Output: pip is just a PyPI package like any other; you could use it to upgrade itself the same way you would upgrade any package: pip install --upgrade pip On Windows the recommended command is: py -m pip install --upgrade pip


Instruction:
List comprehension vs map

Is there a reason to prefer using map() over list comprehension or vice versa? Is either of them generally more efficient or considered generally more Pythonic than the other?

Output: map may be microscopically faster in some cases (when you're not making a lambda for the purpose, but using the same function in map and a list comprehension). List comprehensions may be faster in other cases and most (not all) Pythonistas consider them more direct and clearer. An example of the tiny speed advantage of map when using exactly the same function: $ python -m timeit -s'xs=range(10)' 'map(hex, xs)' 100000 loops, best of 3: 4.86 usec per loop $ python -m timeit -s'xs=range(10)' '[hex(x) for x in xs]' 100000 loops, best of 3: 5.58 usec per loop An example of how performance comparison gets completely reversed when map needs a lambda: $ python -m timeit -s'xs=range(10)' 'map(lambda x: x+2, xs)' 100000 loops, best of 3: 4.24 usec per loop $ python -m timeit -s'xs=range(10)' '[x+2 for x in xs]' 100000 loops, best of 3: 2.32 usec per loop


Instruction:
How can I open multiple files using "with open" in Python?

I want to change a couple of files at one time, iff I can write to all of them. I'm wondering if I somehow can combine the multiple open calls with the with statement: try: with open('a', 'w') as a and open('b', 'w') as b: do_something() except IOError as e: print 'Operation failed: %s' % e.strerror If that's not possible, what would an elegant solution to this problem look like?

Output: As of Python 2.7 (or 3.1 respectively) you can write with open('a', 'w') as a, open('b', 'w') as b: do_something() (Historical note: In earlier versions of Python, you can sometimes use contextlib.nested() to nest context managers. This won't work as expected for opening multiples files, though -- see the linked documentation for details.) In the rare case that you want to open a variable number of files all at the same time, you can use contextlib.ExitStack, starting from Python version 3.3: with ExitStack() as stack: files = [stack.enter_context(open(fname)) for fname in filenames] # Do something with "files" Note that more commonly you want to process files sequentially rather than opening all of them at the same time, in particular if you have a variable number of files: for fname in filenames: with open(fname) as f: # Process f


Instruction:
What does the "at" (@) symbol do in Python?

What does the @ symbol do in Python?

Output: An @ symbol at the beginning of a line is used for class and function decorators: PEP 318: Decorators Python Decorators - Python Wiki The most common Python decorators are: @property @classmethod @staticmethod An @ in the middle of a line is probably matrix multiplication: @ as a binary operator.


Instruction:
What is a clean "pythonic" way to implement multiple constructors?

I can't find a definitive answer for this. As far as I know, you can't have multiple __init__ functions in a Python class. So how do I solve this problem? Suppose I have a class called Cheese with the number_of_holes property. How can I have two ways of creating cheese objects... One that takes a number of holes like this: parmesan = Cheese(num_holes=15). And one that takes no arguments and just randomizes the number_of_holes property: gouda = Cheese(). I can think of only one way to do this, but this seems clunky: class Cheese: def __init__(self, num_holes=0): if num_holes == 0: # Randomize number_of_holes else: number_of_holes = num_holes What do you say? Is there another way?

Output: Actually None is much better for "magic" values: class Cheese: def __init__(self, num_holes=None): if num_holes is None: ... Now if you want complete freedom of adding more parameters: class Cheese: def __init__(self, *args, **kwargs): # args -- tuple of anonymous arguments # kwargs -- dictionary of named arguments self.num_holes = kwargs.get('num_holes', random_holes()) To better explain the concept of *args and **kwargs (you can actually change these names): def f(*args, **kwargs): print('args:', args, 'kwargs:', kwargs) >>> f('a') args: ('a',) kwargs: {} >>> f(ar='a') args: () kwargs: {'ar': 'a'} >>> f(1,2,param=3) args: (1, 2) kwargs: {'param': 3} http://docs.python.org/reference/expressions.html#calls


Instruction:
Find a value in a list

I use the following to check if item is in my_list: if item in my_list: print("Desired item is in list") Is "if item in my_list:" the most "pythonic" way of finding an item in a list? EDIT FOR REOPENING: the question has been considered duplicate, but I'm not entirely convinced: here this question is roughly "what is the most Pythonic way to find an element in a list". And the first answer to the question is really extensive in all Python ways to do this. Whereas on the linked duplicate question and its corresponding answer, the focus is roughly only limited to the 'in' key word in Python. I think it is really limiting, compared to the current question. And I think the answer to this current question, is more relevant and elaborated that the answer of the proposed duplicate question/answer.

Output: As for your first question: "if item is in my_list:" is perfectly fine and should work if item equals one of the elements inside my_list. The item must exactly match an item in the list. For instance, "abc" and "ABC" do not match. Floating point values in particular may suffer from inaccuracy. For instance, 1 - 1/3 != 2/3. As for your second question: There's actually several possible ways if "finding" things in lists. Checking if something is inside This is the use case you describe: Checking whether something is inside a list or not. As you know, you can use the in operator for that: 3 in [1, 2, 3] # => True Filtering a collection That is, finding all elements in a sequence that meet a certain condition. You can use list comprehension or generator expressions for that: matches = [x for x in lst if fulfills_some_condition(x)] matches = (x for x in lst if x > 6) The latter will return a generator which you can imagine as a sort of lazy list that will only be built as soon as you iterate through it. By the way, the first one is exactly equivalent to matches = filter(fulfills_some_condition, lst) in Python 2. Here you can see higher-order functions at work. In Python 3, filter doesn't return a list, but a generator-like object. Finding the first occurrence If you only want the first thing that matches a condition (but you don't know what it is yet), it's fine to use a for loop (possibly using the else clause as well, which is not really well-known). You can also use next(x for x in lst if ...) which will return the first match or raise a StopIteration if none is found. Alternatively, you can use next((x for x in lst if ...), [default value]) Finding the location of an item For lists, there's also the index method that can sometimes be useful if you want to know where a certain element is in the list: [1,2,3].index(2) # => 1 [1,2,3].index(4) # => ValueError However, note that if you have duplicates, .index always returns the lowest index:...... [1,2,3,2].index(2) # => 1 If there are duplicates and you want all the indexes then you can use enumerate() instead: [i for i,x in enumerate([1,2,3,2]) if x==2] # => [1, 3]


Instruction:
python exception message capturing

import ftplib import urllib2 import os import logging logger = logging.getLogger('ftpuploader') hdlr = logging.FileHandler('ftplog.log') formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s') hdlr.setFormatter(formatter) logger.addHandler(hdlr) logger.setLevel(logging.INFO) FTPADDR = "some ftp address" def upload_to_ftp(con, filepath): try: f = open(filepath,'rb') # file to send con.storbinary('STOR '+ filepath, f) # Send the file f.close() # Close file and FTP logger.info('File successfully uploaded to '+ FTPADDR) except, e: logger.error('Failed to upload to ftp: '+ str(e)) This doesn't seem to work, I get syntax error, what is the proper way of doing this for logging all kind of exceptions to a file

Output: You have to define which type of exception you want to catch. So write except Exception as e: instead of except, e: for a general exception. Other possibility is to write your whole try/except code this way: try: with open(filepath,'rb') as f: con.storbinary('STOR '+ filepath, f) logger.info('File successfully uploaded to '+ FTPADDR) except Exception as e: # works on python 3.x logger.error('Failed to upload to ftp: %s', repr(e)) In older versions of Python 2.x, use except Exception, e instead of except Exception as e: try: with open(filepath,'rb') as f: con.storbinary('STOR '+ filepath, f) logger.info('File successfully uploaded to %s', FTPADDR) except Exception, e: # works on python 2.x logger.error('Failed to upload to ftp: %s', repr(e))


Instruction:
How can I iterate over files in a given directory?

I need to iterate through all .asm files inside a given directory and do some actions on them. How can this be done in a efficient way?

Output: Python 3.6 version of the above answer, using os - assuming that you have the directory path as a str object in a variable called directory_in_str: import os directory = os.fsencode(directory_in_str) for file in os.listdir(directory): filename = os.fsdecode(file) if filename.endswith(".asm") or filename.endswith(".py"): # print(os.path.join(directory, filename)) continue else: continue Or recursively, using pathlib: from pathlib import Path pathlist = Path(directory_in_str).glob('**/*.asm') for path in pathlist: # because path is object not string path_in_str = str(path) # print(path_in_str) Use rglob to replace glob('**/*.asm') with rglob('*.asm') This is like calling Path.glob() with '**/' added in front of the given relative pattern: from pathlib import Path pathlist = Path(directory_in_str).rglob('*.asm') for path in pathlist: # because path is object not string path_in_str = str(path) # print(path_in_str) Original answer: import os for filename in os.listdir("/path/to/dir/"): if filename.endswith(".asm") or filename.endswith(".py"): # print(os.path.join(directory, filename)) continue else: continue


Instruction:
Pandas Merging 101

How can I perform a (INNER| (LEFT|RIGHT|FULL) OUTER) JOIN with pandas? How do I add NaNs for missing rows after a merge? How do I get rid of NaNs after merging? Can I merge on the index? How do I merge multiple DataFrames? Cross join with pandas merge? join? concat? update? Who? What? Why?! ... and more. I've seen these recurring questions asking about various facets of the pandas merge functionality. Most of the information regarding merge and its various use cases today is fragmented across dozens of badly worded, unsearchable posts. The aim here is to collate some of the more important points for posterity. This Q&A is meant to be the next installment in a series of helpful user guides on common pandas idioms (see this post on pivoting, and this post on concatenation, which I will be touching on, later). Please note that this post is not meant to be a replacement for the documentation, so please read that as well! Some of the examples are taken from there. Table of Contents For ease of access. Merging basics - basic types of joins (read this first) Index-based joins Generalizing to multiple DataFrames Cross join

Output: This post aims to give readers a primer on SQL-flavored merging with Pandas, how to use it, and when not to use it. In particular, here's what this post will go through: The basics - types of joins (LEFT, RIGHT, OUTER, INNER) merging with different column names merging with multiple columns avoiding duplicate merge key column in output What this post (and other posts by me on this thread) will not go through: Performance-related discussions and timings (for now). Mostly notable mentions of better alternatives, wherever appropriate. Handling suffixes, removing extra columns, renaming outputs, and other specific use cases. There are other (read: better) posts that deal with that, so figure it out! Note Most examples default to INNER JOIN operations while demonstrating various features, unless otherwise specified. Furthermore, all the DataFrames here can be copied and replicated so you can play with them. Also, see this post on how to read DataFrames from your clipboard. Lastly, all visual representation of JOIN operations have been hand-drawn using Google Drawings. Inspiration from here. Enough talk - just show me how to use merge! Setup & Basics np.random.seed(0) left = pd.DataFrame({'key': ['A', 'B', 'C', 'D'], 'value': np.random.randn(4)}) right = pd.DataFrame({'key': ['B', 'D', 'E', 'F'], 'value': np.random.randn(4)}) left key value 0 A 1.764052 1 B 0.400157 2 C 0.978738 3 D 2.240893 right key value 0 B 1.867558 1 D -0.977278 2 E 0.950088 3 F -0.151357 For the sake of simplicity, the key column has the same name (for now). An INNER JOIN is represented by Note This, along with the forthcoming figures all follow this convention: blue indicates rows that are present in the merge result red indicates rows that are excluded from the result (i.e., removed) green indicates missing values that are replaced with NaNs in the result To perform an INNER JOIN, call merge on the left DataFrame, specifying the right DataFrame and the join key (at the very least) as arguments. left.merge(right, on='key') # Or, if you want to be explicit # left.merge(right, on='key', how='inner') key value_x value_y 0 B 0.400157 1.867558 1 D 2.240893 -0.977278 This returns only rows from left and right which share a common key (in this example, "B" and "D). A LEFT OUTER JOIN, or LEFT JOIN is represented by This can be performed by specifying how='left'. left.merge(right, on='key', how='left') key value_x value_y 0 A 1.764052 NaN 1 B 0.400157 1.867558 2 C 0.978738 NaN 3 D 2.240893 -0.977278 Carefully note the placement of NaNs here. If you specify how='left', then only keys from left are used, and missing data from right is replaced by NaN. And similarly, for a RIGHT OUTER JOIN, or RIGHT JOIN which is... ...specify how='right': left.merge(right, on='key', how='right') key value_x value_y 0 B 0.400157 1.867558 1 D 2.240893 -0.977278 2 E NaN 0.950088 3 F NaN -0.151357 Here, keys from right are used, and missing data from left is replaced by NaN. Finally, for the FULL OUTER JOIN, given by specify how='outer'. left.merge(right, on='key', how='outer') key value_x value_y 0 A 1.764052 NaN 1 B 0.400157 1.867558 2 C 0.978738 NaN 3 D 2.240893 -0.977278 4 E NaN 0.950088 5 F NaN -0.151357 This uses the keys from both frames, and NaNs are inserted for missing rows in both. The documentation summarizes these various merges nicely: Other JOINs - LEFT-Excluding, RIGHT-Excluding, and FULL-Excluding/ANTI JOINs If you need LEFT-Excluding JOINs and RIGHT-Excluding JOINs in two steps. For LEFT-Excluding JOIN, represented as Start by performing a LEFT OUTER JOIN and then filtering to rows coming from left only (excluding everything from the right), (left.merge(right, on='key', how='left', indicator=True) .query('_merge == "left_only"') .drop('_merge', axis=1)) key value_x value_y 0 A 1.764052 NaN 2 C 0.978738 NaN Where, left.merge(right, on='key', how='left', indicator=True) key value_x value_y _merge 0 A 1.764052 NaN left_only 1 B 0.400157 1.867558 both 2 C 0.978738 NaN left_only 3 D 2.240893 -0.977278 both And similarly, for a RIGHT-Excluding JOIN, (left.merge(right, on='key', how='right', indicator=True) .query('_merge == "right_only"') .drop('_merge', axis=1)) key value_x value_y 2 E NaN 0.950088 3 F NaN -0.151357 Lastly, if you are required to do a merge that only retains keys from the left or right, but not both (IOW, performing an ANTI-JOIN), You can do this in similar fashion— (left.merge(right, on='key', how='outer', indicator=True) .query('_merge != "both"') .drop('_merge', axis=1)) key value_x value_y 0 A 1.764052 NaN 2 C 0.978738 NaN 4 E NaN 0.950088 5 F NaN -0.151357 Different names for key columns If the key columns are named differently—for example, left has keyLeft, and right has keyRight instead of key—then you will have to specify left_on and right_on as arguments instead of on: left2 = left.rename({'key':'keyLeft'}, axis=1) right2 = right.rename({'key':'keyRight'}, axis=1) left2 keyLeft value 0 A 1.764052 1 B 0.400157 2 C 0.978738 3 D 2.240893 right2 keyRight value 0 B 1.867558 1 D -0.977278 2 E 0.950088 3 F -0.151357 left2.merge(right2, left_on='keyLeft', right_on='keyRight', how='inner') keyLeft value_x keyRight value_y 0 B 0.400157 B 1.867558 1 D 2.240893 D -0.977278 Avoiding duplicate key column in output When merging on keyLeft from left and keyRight from right, if you only want either of the keyLeft or keyRight (but not both) in the output, you can start by setting the index as a preliminary step. left3 = left2.set_index('keyLeft') left3.merge(right2, left_index=True, right_on='keyRight') value_x keyRight value_y 0 0.400157 B 1.867558 1 2.240893 D -0.977278 Contrast this with the output of the command just before (that is, the output of left2.merge(right2, left_on='keyLeft', right_on='keyRight', how='inner')), you'll notice keyLeft is missing. You can figure out what column to keep based on which frame's index is set as the key. This may matter when, say, performing some OUTER JOIN operation. Merging only a single column from one of the DataFrames For example, consider right3 = right.assign(newcol=np.arange(len(right))) right3 key value newcol 0 B 1.867558 0 1 D -0.977278 1 2 E 0.950088 2 3 F -0.151357 3 If you are required to merge only "newcol" (without any of the other columns), you can usually just subset columns before merging: left.merge(right3[['key', 'newcol']], on='key') key value newcol 0 B 0.400157 0 1 D 2.240893 1 If you're doing a LEFT OUTER JOIN, a more performant solution would involve map: # left['newcol'] = left['key'].map(right3.set_index('key')['newcol'])) left.assign(newcol=left['key'].map(right3.set_index('key')['newcol'])) key value newcol 0 A 1.764052 NaN 1 B 0.400157 0.0 2 C 0.978738 NaN 3 D 2.240893 1.0 As mentioned, this is similar to, but faster than left.merge(right3[['key', 'newcol']], on='key', how='left') key value newcol 0 A 1.764052 NaN 1 B 0.400157 0.0 2 C 0.978738 NaN 3 D 2.240893 1.0 Merging on multiple columns To join on more than one column, specify a list for on (or left_on and right_on, as appropriate). left.merge(right, on=['key1', 'key2'] ...) Or, in the event the names are different, left.merge(right, left_on=['lkey1', 'lkey2'], right_on=['rkey1', 'rkey2']) Other useful merge* operations and functions Merging a DataFrame with Series on index: See this answer. Besides merge, DataFrame.update and DataFrame.combine_first are also used in certain cases to update one DataFrame with another. pd.merge_ordered is a useful function for ordered JOINs. pd.merge_asof (read: merge_asOf) is useful for approximate joins. This section only covers the very basics, and is designed to only whet your appetite. For more examples and cases, see the documentation on merge, join, and concat as well as the links to the function specifications. Continue Reading Jump to other topics in Pandas Merging 101 to continue learning: Merging basics - basic types of joins * Index-based joins Generalizing to multiple DataFrames Cross join *You are here.


Instruction:
Python setup.py uninstall

I have installed a Python package with python setup.py install. How do I uninstall it?

Output: Note: Avoid using python setup.py install. Instead, use pip install . You need to remove all files manually, and also undo any other stuff that installation did manually. If you don't know the list of all files, you can reinstall it with the --record option, and take a look at the list this produces. To record a list of installed files, you can use: python setup.py install --record files.txt Once you want to uninstall you can use xargs to do the removal: xargs rm -rf < files.txt Or if you're running Windows, use PowerShell: Get-Content files.txt | ForEach-Object {Remove-Item $_ -Recurse -Force} Then delete also the containing directory, e.g. /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/my_module-0.1.egg/ on macOS. It doesn't have any files, but Python will still import an empty module: >>> import my_module >>> my_module.__file__ None Once deleted, Python shows: >>> import my_module Traceback (most recent call last): File "<stdin>", line 1, in <module> ModuleNotFoundError: No module named 'my_module'


Instruction:
What is the best "not None" test in Python

Out of these not None tests. if val != None: if not (val is None): if val is not None: Which one is preferable, and why?

Output: if val is not None: # ... is the Pythonic idiom for testing that a variable is not set to None. This idiom has particular uses in the case of declaring keyword functions with default parameters. is tests identity in Python. Because there is one and only one instance of None present in a running Python script/program, is is the optimal test for this. As Johnsyweb points out, this is discussed in PEP 8 under "Programming Recommendations". As for why this is preferred to if not (val is None): # ... this is simply part of the Zen of Python: "Readability counts." Good Python is often close to good pseudocode.


Instruction:
Get the last day of the month

Can Python's standard library (or dateutil if not) easily determine (i.e. one function call) the last day of a given month?

Output: calendar.monthrange provides this information: calendar.monthrange(year, month) Returns weekday of first day of the month and number of days in month, for the specified year and month. >>> import calendar >>> calendar.monthrange(2008, 2) # leap years are handled correctly (calendar.FRIDAY, 29) >>> calendar.monthrange(2100, 2) # years divisible by 100 but not 400 aren't leap years (calendar.MONDAY, 28) # the module uses the Georgian calendar extended into the past and # future, so leap days in the distant past will differ from Julian: >>> calendar.monthrange(1100, 2) (calendar.THURSDAY, 28) # note also that pre-Python 3.12, the REPL renders the weekday # as a bare integer: >>> calendar.monthrange(2002, 1) (1, 31) so simply: calendar.monthrange(year, month)[1]


Instruction:
How to change the font size on a matplotlib plot

How does one change the font size for all elements (ticks, labels, title) on a matplotlib plot? I know how to change the tick label sizes, this is done with: import matplotlib matplotlib.rc('xtick', labelsize=20) matplotlib.rc('ytick', labelsize=20) But how does one change the rest?

Output: From the matplotlib documentation, font = {'family' : 'normal', 'weight' : 'bold', 'size' : 22} matplotlib.rc('font', **font) This sets the font of all items to the font specified by the kwargs object, font. Alternatively, you could also use the rcParams update method as suggested in this answer: matplotlib.rcParams.update({'font.size': 22}) or import matplotlib.pyplot as plt plt.rcParams.update({'font.size': 22}) You can find a full list of available properties on the Customizing matplotlib page.


Instruction:
How to remove items from a list while iterating?

I'm iterating over a list of tuples in Python, and am attempting to remove them if they meet certain criteria. for tup in somelist: if determine(tup): code_to_remove_tup What should I use in place of code_to_remove_tup? I can't figure out how to remove the item in this fashion.

Output: You can use a list comprehension to create a new list containing only the elements you don't want to remove: somelist = [x for x in somelist if not determine(x)] Or, by assigning to the slice somelist[:], you can mutate the existing list to contain only the items you want: somelist[:] = [x for x in somelist if not determine(x)] This approach could be useful if there are other references to somelist that need to reflect the changes. Instead of a comprehension, you could also use itertools. In Python 2: from itertools import ifilterfalse somelist[:] = ifilterfalse(determine, somelist) Or in Python 3: from itertools import filterfalse somelist[:] = filterfalse(determine, somelist)


Instruction:
Create an empty list with certain size in Python

How do I create an empty list that can hold 10 elements? After that, I want to assign values in that list. For example: xs = list() for i in range(0, 9): xs[i] = i However, that gives IndexError: list assignment index out of range. Why?

Output: You cannot assign to a list like xs[i] = value, unless the list already is initialized with at least i+1 elements (because the first index is 0). Instead, use xs.append(value) to add elements to the end of the list. (Though you could use the assignment notation if you were using a dictionary instead of a list.) Creating an empty list: >>> xs = [None] * 10 >>> xs [None, None, None, None, None, None, None, None, None, None] Assigning a value to an existing element of the above list: >>> xs[1] = 5 >>> xs [None, 5, None, None, None, None, None, None, None, None] Keep in mind that something like xs[15] = 5 would still fail, as our list has only 10 elements. range(x) creates a list from [0, 1, 2, ... x-1] # 2.X only. Use list(range(10)) in 3.X. >>> xs = range(10) >>> xs [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] Using a function to create a list: >>> def display(): ... xs = [] ... for i in range(9): # This is just to tell you how to create a list. ... xs.append(i) ... return xs ... >>> print display() [0, 1, 2, 3, 4, 5, 6, 7, 8] List comprehension (Using the squares because for range you don't need to do all this, you can just return range(0,9) ): >>> def display(): ... return [x**2 for x in range(9)] ... >>> print display() [0, 1, 4, 9, 16, 25, 36, 49, 64]


Instruction:
How to convert string representation of list to a list

I was wondering what the simplest way is to convert a string representation of a list like the following to a list: x = '[ "A","B","C" , " D"]' Even in cases where the user puts spaces in between the commas, and spaces inside of the quotes, I need to handle that as well and convert it to: x = ["A", "B", "C", "D"] I know I can strip spaces with strip() and split() and check for non-letter characters. But the code was getting very kludgy. Is there a quick function that I'm not aware of?

Output: >>> import ast >>> x = '[ "A","B","C" , " D"]' >>> x = ast.literal_eval(x) >>> x ['A', 'B', 'C', ' D'] >>> x = [n.strip() for n in x] >>> x ['A', 'B', 'C', 'D'] ast.literal_eval: Evaluate an expression node or a string containing only a Python literal or container display. The string or node provided may only consist of the following Python literal structures: strings, bytes, numbers, tuples, lists, dicts, sets, booleans, None and Ellipsis. This can be used for evaluating strings containing Python values without the need to parse the values oneself. It is not capable of evaluating arbitrarily complex expressions, for example involving operators or indexing.


Instruction:
What exactly do "u" and "r" string prefixes do, and what are raw string literals?

While asking this question, I realized I didn't know much about raw strings. For somebody claiming to be a Django trainer, this sucks. I know what an encoding is, and I know what u'' alone does since I get what is Unicode. But what does r'' do exactly? What kind of string does it result in? And above all, what the heck does ur'' do? Finally, is there any reliable way to go back from a Unicode string to a simple raw string? Ah, and by the way, if your system and your text editor charset are set to UTF-8, does u'' actually do anything?

Output: There's not really any "raw string"; there are raw string literals, which are exactly the string literals marked by an r before the opening quote. A "raw string literal" is a slightly different syntax for a string literal, in which a backslash, \, is taken as meaning "just a backslash" (except when it comes right before a quote that would otherwise terminate the literal) -- no "escape sequences" to represent newlines, tabs, backspaces, form-feeds, and so on. In normal string literals, each backslash must be doubled up to avoid being taken as the start of an escape sequence. This syntax variant exists mostly because the syntax of regular expression patterns is heavy with backslashes (but never at the end, so the "except" clause above doesn't matter) and it looks a bit better when you avoid doubling up each of them -- that's all. It also gained some popularity to express native Windows file paths (with backslashes instead of regular slashes like on other platforms), but that's very rarely needed (since normal slashes mostly work fine on Windows too) and imperfect (due to the "except" clause above). r'...' is a byte string (in Python 2.*), ur'...' is a Unicode string (again, in Python 2.*), and any of the other three kinds of quoting also produces exactly the same types of strings (so for example r'...', r'''...''', r"...", r"""...""" are all byte strings, and so on). Not sure what you mean by "going back" - there is no intrinsically back and forward directions, because there's no raw string type, it's just an alternative syntax to express perfectly normal string objects, byte or unicode as they may be. And yes, in Python 2.*, u'...' is of course always distinct from just '...' -- the former is a unicode string, the latter is a byte string. What encoding the literal might be expressed in is a completely orthogonal issue. E.g., consider (Python 2.6): >>> sys.getsizeof('ciao') 28 >>> sys.getsizeof(u'ciao') 34 The Unicode object of course takes more memory space (very small difference for a very short string, obviously ;-).


Instruction:
How do I print the full NumPy array, without truncation?

When I print a numpy array, I get a truncated representation, but I want the full array. >>> numpy.arange(10000) array([ 0, 1, 2, ..., 9997, 9998, 9999]) >>> numpy.arange(10000).reshape(250,40) array([[ 0, 1, 2, ..., 37, 38, 39], [ 40, 41, 42, ..., 77, 78, 79], [ 80, 81, 82, ..., 117, 118, 119], ..., [9880, 9881, 9882, ..., 9917, 9918, 9919], [9920, 9921, 9922, ..., 9957, 9958, 9959], [9960, 9961, 9962, ..., 9997, 9998, 9999]])

Output: Use numpy.set_printoptions: import sys import numpy numpy.set_printoptions(threshold=sys.maxsize)


Instruction:
Creating an empty Pandas DataFrame, and then filling it

I'm starting from the pandas DataFrame documentation here: Introduction to data structures I'd like to iteratively fill the DataFrame with values in a time series kind of calculation. I'd like to initialize the DataFrame with columns A, B, and timestamp rows, all 0 or all NaN. I'd then add initial values and go over this data calculating the new row from the row before, say row[A][t] = row[A][t-1]+1 or so. I'm currently using the code as below, but I feel it's kind of ugly and there must be a way to do this with a DataFrame directly or just a better way in general. import pandas as pd import datetime as dt import scipy as s base = dt.datetime.today().date() dates = [ base - dt.timedelta(days=x) for x in range(9, -1, -1) ] valdict = {} symbols = ['A','B', 'C'] for symb in symbols: valdict[symb] = pd.Series( s.zeros(len(dates)), dates ) for thedate in dates: if thedate > dates[0]: for symb in valdict: valdict[symb][thedate] = 1 + valdict[symb][thedate - dt.timedelta(days=1)]

Output: Here's a couple of suggestions: Use date_range for the index: import datetime import pandas as pd import numpy as np todays_date = datetime.datetime.now().date() index = pd.date_range(todays_date-datetime.timedelta(10), periods=10, freq='D') columns = ['A','B', 'C'] Note: we could create an empty DataFrame (with NaNs) simply by writing: df_ = pd.DataFrame(index=index, columns=columns) df_ = df_.fillna(0) # With 0s rather than NaNs To do these type of calculations for the data, use a NumPy array: data = np.array([np.arange(10)]*3).T Hence we can create the DataFrame: In [10]: df = pd.DataFrame(data, index=index, columns=columns) In [11]: df Out[11]: A B C 2012-11-29 0 0 0 2012-11-30 1 1 1 2012-12-01 2 2 2 2012-12-02 3 3 3 2012-12-03 4 4 4 2012-12-04 5 5 5 2012-12-05 6 6 6 2012-12-06 7 7 7 2012-12-07 8 8 8 2012-12-08 9 9 9


Instruction:
Truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all()

I want to filter my dataframe with an or condition to keep rows with a particular column's values that are outside the range [-0.25, 0.25]. I tried: df = df[(df['col'] < -0.25) or (df['col'] > 0.25)] But I get the error: ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

Output: The or and and Python statements require truth-values. For pandas, these are considered ambiguous, so you should use "bitwise" | (or) or & (and) operations: df = df[(df['col'] < -0.25) | (df['col'] > 0.25)] These are overloaded for these kinds of data structures to yield the element-wise or or and. Just to add some more explanation to this statement: The exception is thrown when you want to get the bool of a pandas.Series: >>> import pandas as pd >>> x = pd.Series([1]) >>> bool(x) ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all(). You hit a place where the operator implicitly converted the operands to bool (you used or but it also happens for and, if and while): >>> x or x ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all(). >>> x and x ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all(). >>> if x: ... print('fun') ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all(). >>> while x: ... print('fun') ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all(). Besides these four statements, there are several Python functions that hide some bool calls (like any, all, filter, ...). These are normally not problematic with pandas.Series, but for completeness I wanted to mention these. In your case, the exception isn't really helpful, because it doesn't mention the right alternatives. For and and or, if you want element-wise comparisons, you can use: numpy.logical_or: >>> import numpy as np >>> np.logical_or(x, y) or simply the | operator: >>> x | y numpy.logical_and: >>> np.logical_and(x, y) or simply the & operator: >>> x & y If you're using the operators, then be sure to set your parentheses correctly because of operator precedence. There are several logical NumPy functions which should work on pandas.Series. The alternatives mentioned in the Exception are more suited if you encountered it when doing if or while. I'll shortly explain each of these: If you want to check if your Series is empty: >>> x = pd.Series([]) >>> x.empty True >>> x = pd.Series([1]) >>> x.empty False Python normally interprets the length of containers (like list, tuple, ...) as truth-value if it has no explicit Boolean interpretation. So if you want the Python-like check, you could do: if x.size or if not x.empty instead of if x. If your Series contains one and only one Boolean value: >>> x = pd.Series([100]) >>> (x > 50).bool() True >>> (x < 50).bool() False If you want to check the first and only item of your Series (like .bool(), but it works even for non-Boolean contents): >>> x = pd.Series([100]) >>> x.item() 100 If you want to check if all or any item is not-zero, not-empty or not-False: >>> x = pd.Series([0, 1, 2]) >>> x.all() # Because one element is zero False >>> x.any() # because one (or more) elements are non-zero True