Instruction:
Which Python packages offer a stand-alone event system?

I am aware of pydispatcher, but there must be other event-related packages around for Python. Which libraries are available? I'm not interested in event managers that are part of large frameworks, I'd rather use a small bare-bones solution that I can easily extend.

Output: PyPI packages As of February 2025, these are the event-related packages available on PyPI, ordered by most recent release date. psygnal 0.12.0: February 2024 pymitter 1.0.0: January 2025 blinker 1.9.0: November 2024 pluggy 1.5.0: April 2024 Louie 2.0.1: July 2023 Events 0.5: July 2023 2020 zope.event 5.0: June 2023 PyDispatcher 2.0.7: February 2023 python-dispatch 0.2.2: Jun 2023 python-dispatch 0.2.2: Jun 2023 RxPy3 1.0.1: June 2020 PyPubSub 4.0.3: Jan 2019 pyeventdispatcher 0.2.3a0: 2018 buslane 0.0.5: 2018 PyPyDispatcher 2.1.2: 2017 axel 0.0.7: 2016 dispatcher 1.0: 2012 py-notify 0.3.1: 2008 There's more That's a lot of libraries to choose from, using very different terminology (events, signals, handlers, method dispatch, hooks, ...). I'm trying to keep an overview of the above packages, plus the techniques mentioned in the answers here. First, some terminology... Observer pattern The most basic style of event system is the 'bag of handler methods', which is a simple implementation of the Observer pattern. Basically, the handler methods (callables) are stored in an array and are each called when the event 'fires'. Publish-Subscribe The disadvantage of Observer event systems is that you can only register the handlers on the actual Event object (or handlers list). So at registration time the event already needs to exist. That's why the second style of event systems exists: the publish-subscribe pattern. Here, the handlers don't register on an event object (or handler list), but on a central dispatcher. Also the notifiers only talk to the dispatcher. What to listen for, or what to publish is determined by 'signal', which is nothing more than a name (string). Mediator pattern Might be of interest as well: the Mediator pattern. Hooks A 'hook' system is usally used in the context of application plugins. The application contains fixed integration points (hooks), and each plugin may connect to that hook and perform certain actions. Other 'events' Note: threading.Event is not an 'event system' in the above sense. It's a thread synchronization system where one thread waits until another thread 'signals' the Event object. Network messaging libraries often use the term 'events' too; sometimes these are similar in concept; sometimes not. They can of course traverse thread-, process- and computer boundaries. See e.g. pyzmq, pymq, Twisted, Tornado, gevent, eventlet. Weak references In Python, holding a reference to a method or object ensures that it won't get deleted by the garbage collector. This can be desirable, but it can also lead to memory leaks: the linked handlers are never cleaned up. Some event systems use weak references instead of regular ones to solve this. Some words about the various libraries Observer-style event systems: psygnal has a very clean interface with connect() and emit() methods. zope.event shows the bare bones of how this works (see Lennart's answer). Note: this example does not even support handler arguments. LongPoke's 'callable list' implementation shows that such an event system can be implemented very minimalistically by subclassing list. Felk's variation EventHook also ensures the signatures of callees and callers. spassig's EventHook (Michael Foord's Event Pattern) is a straightforward implementation. Josip's Valued Lessons Event class is basically the same, but uses a set instead of a list to store the bag, and implements __call__ which are both reasonable additions. PyNotify is similar in concept and also provides additional concepts of variables and conditions ('variable changed event'). Homepage is not functional. axel is basically a bag-of-handlers with more features related to threading, error handling, ... python-dispatch requires the even source classes to derive from pydispatch.Dispatcher. buslane is class-based, supports single- or multiple handlers and facilitates extensive type hints. Pithikos' Observer/Event is a lightweight design. Publish-subscribe libraries: blinker has some nifty features such as automatic disconnection and filtering based on sender. PyPubSub is a stable package, and promises "advanced features that facilitate debugging and maintaining topics and messages". pymitter is a Python port of Node.js EventEmitter2 and offers namespaces, wildcards and TTL. PyDispatcher seems to emphasize flexibility with regards to many-to-many publication etc. Supports weak references. louie is a reworked PyDispatcher and should work "in a wide variety of contexts". pypydispatcher is based on (you guessed it...) PyDispatcher and also works in PyPy. django.dispatch is a rewritten PyDispatcher "with a more limited interface, but higher performance". pyeventdispatcher is based on PHP's Symfony framework's event-dispatcher. dispatcher was extracted from django.dispatch but is getting fairly old. Cristian Garcia's EventManger is a really short implementation. Others: pluggy contains a hook system which is used by pytest plugins. RxPy3 implements the Observable pattern and allows merging events, retry etc. Qt's Signals and Slots are available from PyQt or PySide2. They work as callback when used in the same thread, or as events (using an event loop) between two different threads. Signals and Slots have the limitation that they only work in objects of classes that derive from QObject.


Instruction:
Writing a Python list of lists to a csv file

I have a long list of lists of the following form --- a = [[1.2,'abc',3],[1.2,'werew',4],........,[1.4,'qew',2]] i.e. the values in the list are of different types -- float,int, strings.How do I write it into a csv file so that my output csv file looks like 1.2,abc,3 1.2,werew,4 . . . 1.4,qew,2

Output: Python's built-in csv module can handle this easily: import csv with open('out.csv', 'w', newline='') as f: writer = csv.writer(f) writer.writerows(a) This assumes your list is defined as a, as it is in your question. You can tweak the exact format of the output CSV via the various optional parameters to csv.writer().


Instruction:
Iterating through directories with Python

I need to iterate through the subdirectories of a given directory and search for files. If I get a file I have to open it and change the content and replace it with my own lines. I tried this: import os rootdir ='C:/Users/sid/Desktop/test' for subdir, dirs, files in os.walk(rootdir): for file in files: f=open(file,'r') lines=f.readlines() f.close() f=open(file,'w') for line in lines: newline = "No you are not" f.write(newline) f.close() but I am getting an error. What am I doing wrong?

Output: The actual walk through the directories works as you have coded it. If you replace the contents of the inner loop with a simple print statement you can see that each file is found: import os rootdir = 'C:/Users/sid/Desktop/test' for subdir, dirs, files in os.walk(rootdir): for file in files: print(os.path.join(subdir, file))


Instruction:
What does |= (ior) do in Python?

Google won't let me search |= so I'm having trouble finding relevant documentation. Anybody know?

Output: |= performs an in-place+ operation between pairs of objects. In particular, between: sets: a union operation dicts: an update operation counters: a union (of multisets) operation numbers: a bitwise OR, binary operation In most cases, it is related to the | operator. See examples below. Sets For example, the union of two sets assigned to s1 and s2 share the following equivalent expressions: >>> s1 = s1 | s2 # 1 >>> s1 |= s2 # 2 >>> s1.__ior__(s2) # 3 where the final value of s1 is equivalent either by: an assigned OR operation an in-place OR operation an in-place OR operation via special method++ Example Here we apply OR (|) and the in-place OR (|=) to sets: >>> s1 = {"a", "b", "c"} >>> s2 = {"d", "e", "f"} >>> # OR, | >>> s1 | s2 {'a', 'b', 'c', 'd', 'e', 'f'} >>> s1 # `s1` is unchanged {'a', 'b', 'c'} >>> # In-place OR, |= >>> s1 |= s2 >>> s1 # `s1` is reassigned {'a', 'b', 'c', 'd', 'e', 'f'} Dictionaries In Python 3.9+, new merge (|) and update (|=) operators are proposed between dictionaries. Note: these are not the same as set operators mentioned above. Given operations between two dicts assigned to d1 and d2: >>> d1 = d1 | d2 # 1 >>> d1 |= d2 # 2 where d1 is equivalent via: an assigned merge-right operation an in-place merge-right (update) operation; equivalent to d1.update(d2) Example Here we apply merge (|) and update (|=) to dicts: >>> d1 = {"a": 0, "b": 1, "c": 2} >>> d2 = {"c": 20, "d": 30} >>> # Merge, | >>> d1 | d2 {"a": 0, "b": 1, "c": 20, "d": 30} >>> d1 {"a": 0, "b": 1, "c": 2} >>> # Update, |= >>> d1 |= d2 >>> d1 {"a": 0, "b": 1, "c": 20, "d": 30} Counters The collections.Counter is related to a mathematical datastructure called a multiset (mset). It is basically a dict of (object, multiplicity) key-value pairs. Given operations between two counters assigned to c1 and c2: >>> c1 = c1 | c2 # 1 >>> c1 |= c2 # 2 where c1 is equivalent via: an assigned union operation an in-place union operation A union of multisets contains the maximum multiplicities per entry. Note, this does not behave the same way as between two sets or between two regular dicts. Example Here we apply union (|) and the in-place union (|=) to Counters: import collections as ct >>> c1 = ct.Counter({2: 2, 3: 3}) >>> c2 = ct.Counter({1: 1, 3: 5}) >>> # Union, | >>> c1 | c2 Counter({2: 2, 3: 5, 1: 1}) >>> c1 Counter({2: 2, 3: 3}) >>> # In-place Union, |= >>> c1 |= c2 >>> c1 Counter({2: 2, 3: 5, 1: 1}) Numbers Lastly, you can do binary math. Given operations between two numbers assigned to n1 and n2: >>> n1 = n1 | n2 # 1 >>> n1 |= n2 # 2 where n1 is equivalent via: an assigned bitwise OR operation an in-place bitwise OR operation Example Here we apply bitwise OR (|) and the in-place bitwise OR (|=) to numbers: >>> n1 = 0 >>> n2 = 1 >>> # Bitwise OR, | >>> n1 | n2 1 >>> n1 0 >>> # In-place Bitwise OR, |= >>> n1 |= n2 >>> n1 1 Review This section briefly reviews some bitwise math. In the simplest case, the bitwise OR operation compares two binary bits. It will always return 1 except when both bits are 0. >>> assert 1 == (1 | 1) == (1 | 0) == (0 | 1) >>> assert 0 == (0 | 0) We now extend this idea beyond binary numbers. Given any two integral numbers (lacking fractional components), we apply the bitwise OR and get an integral result: >>> a = 10 >>> b = 16 >>> a | b 26 How? In general, the bitwise operations follow some "rules": internally compare binary equivalents apply the operation return the result as the given type Let's apply these rules to our regular integers above. (1) Compare binary equivalents, seen here as strings (0b denotes binary): >>> bin(a) '0b1010' >>> bin(b) '0b10000' (2) Apply a bitwise OR operation to each column (0 when both are 0, else 1): 01010 10000 ----- 11010 (3) Return the result in the given type, e.g. base 10, decimal: >>> int(0b11010) 26 The internal binary comparison means we can apply the latter to integers in any base, e.g. hex and octal: >>> a = 10 # 10, dec >>> b = 0b10000 # 16, bin >>> c = 0xa # 10, hex >>> d = 0o20 # 16, oct >>> a | b 26 >>> c | d 26 See Also An example of overloading the __ior__() method to iterate iterables in a MutableSet abstract base class R. Hettinger's OrderedSet recipe (see lines 3 and 10 respectively) A thread on Python-ideas on why to use |= to update a set A section B.8 of Dive in Python 3 on special methods of Python operators In-place binary operators fallback to regular methods, see cpython source code (eval.c and abstract.c). Thanks @asottile. A post on how python handles displaying prepended zeros in bitwise computations +The in-place bitwise OR operator cannot be applied to literals; assign objects to names. ++Special methods return the same operations as their corresponding operators.


Instruction:
Suppress Scientific Notation in Numpy When Creating Array From Nested List

I have a nested Python list that looks like the following: my_list = [[3.74, 5162, 13683628846.64, 12783387559.86, 1.81], [9.55, 116, 189688622.37, 260332262.0, 1.97], [2.2, 768, 6004865.13, 5759960.98, 1.21], [3.74, 4062, 3263822121.39, 3066869087.9, 1.93], [1.91, 474, 44555062.72, 44555062.72, 0.41], [5.8, 5006, 8254968918.1, 7446788272.74, 3.25], [4.5, 7887, 30078971595.46, 27814989471.31, 2.18], [7.03, 116, 66252511.46, 81109291.0, 1.56], [6.52, 116, 47674230.76, 57686991.0, 1.43], [1.85, 623, 3002631.96, 2899484.08, 0.64], [13.76, 1227, 1737874137.5, 1446511574.32, 4.32], [13.76, 1227, 1737874137.5, 1446511574.32, 4.32]] I then import Numpy, and set print options to (suppress=True). When I create an array: my_array = numpy.array(my_list) I can't for the life of me suppress scientific notation: [[ 3.74000000e+00 5.16200000e+03 1.36836288e+10 1.27833876e+10 1.81000000e+00] [ 9.55000000e+00 1.16000000e+02 1.89688622e+08 2.60332262e+08 1.97000000e+00] [ 2.20000000e+00 7.68000000e+02 6.00486513e+06 5.75996098e+06 1.21000000e+00] [ 3.74000000e+00 4.06200000e+03 3.26382212e+09 3.06686909e+09 1.93000000e+00] [ 1.91000000e+00 4.74000000e+02 4.45550627e+07 4.45550627e+07 4.10000000e-01] [ 5.80000000e+00 5.00600000e+03 8.25496892e+09 7.44678827e+09 3.25000000e+00] [ 4.50000000e+00 7.88700000e+03 3.00789716e+10 2.78149895e+10 2.18000000e+00] [ 7.03000000e+00 1.16000000e+02 6.62525115e+07 8.11092910e+07 1.56000000e+00] [ 6.52000000e+00 1.16000000e+02 4.76742308e+07 5.76869910e+07 1.43000000e+00] [ 1.85000000e+00 6.23000000e+02 3.00263196e+06 2.89948408e+06 6.40000000e-01] [ 1.37600000e+01 1.22700000e+03 1.73787414e+09 1.44651157e+09 4.32000000e+00] [ 1.37600000e+01 1.22700000e+03 1.73787414e+09 1.44651157e+09 4.32000000e+00]] If I create a simple numpy array directly: new_array = numpy.array([1.5, 4.65, 7.845]) I have no problem and it prints as follows: [ 1.5 4.65 7.845] Does anyone know what my problem is?

Output: This is what you need: np.set_printoptions(suppress=True) Here is the documentation which says suppress: bool, optional If True, always print floating point numbers using fixed point notation, in which case numbers equal to zero in the current precision will print as zero. If False, then scientific notation is used when absolute value of the smallest number is < 1e-4 or the ratio of the maximum absolute value to the minimum is > 1e3. The default is False. In the original question, the difference between the array created "directly" and the original "big" array is that the big array contains very large numbers (e.g. 1.44651157e+09), so NumPy chooses the scientific notation for it, unless it's suppressed.


Instruction:
In Python, how does one catch warnings as if they were exceptions?

A third-party library (written in C) that I use in my python code is issuing warnings. I want to be able to use the try except syntax to properly handle these warnings. Is there a way to do this?

Output: To handle warnings as errors simply use this: import warnings warnings.filterwarnings("error") After this you will be able to catch warnings same as errors, e.g. this will work: try: some_heavy_calculations() except RuntimeWarning: breakpoint() You can also reset the behaviour of warnings by running: warnings.resetwarnings() P.S. Added this answer because the best answer in comments contains misspelling: filterwarnigns instead of filterwarnings.


Instruction:
assertEquals vs. assertEqual in python

Is there a difference between assertEquals and assertEqual in the python unittest.TestCase? And if there is not, why are there two functions? Only for convenience?

Output: Actually, in Python 2.6, both assertEqual and assertEquals are convenience aliases to failUnlessEqual. The source declares them thus: # Synonyms for assertion methods assertEqual = assertEquals = failUnlessEqual In Python 3, to your point, failUnlessEqual is explicitly deprecated. assertEquals carries this comment :-) # Synonyms for assertion methods # The plurals are undocumented. Keep them that way to discourage use. # Do not add more. Do not remove. # Going through a deprecation cycle on these would annoy many people. So, the upshot appears to be that you should use whatever you like for Python 2.x, but tend toward assertEqual for Python 3.


Instruction:
How can I calculate a rolling / moving average using Python + NumPy / SciPy?

There doesn’t seem to be any function in NumPy or SciPy that simply calculate the moving average, leading to convoluted solutions. My question is twofold: What's the easiest way to (correctly) implement a moving average with NumPy? Since this seems nontrivial and error-prone, is there a good reason not to have the batteries included in this case?

Output: A simple way to achieve this is by using np.convolve. The idea behind this is to leverage the way the discrete convolution is computed and use it to return a rolling mean. This can be done by convolving with a sequence of np.ones of a length equal to the sliding window length we want. In order to do so we could define the following function: def moving_average(x, w): return np.convolve(x, np.ones(w), 'valid') / w This function will be taking the convolution of the sequence x and a sequence of ones of length w. Note that the chosen mode is valid so that the convolution product is only given for points where the sequences overlap completely. Some examples: x = np.array([5,3,8,10,2,1,5,1,0,2]) For a moving average with a window of length 2 we would have: moving_average(x, 2) # array([4. , 5.5, 9. , 6. , 1.5, 3. , 3. , 0.5, 1. ]) And for a window of length 4: moving_average(x, 4) # array([6.5 , 5.75, 5.25, 4.5 , 2.25, 1.75, 2. ]) How does convolve work? Let’s have a more in-depth look at the way the discrete convolution is being computed. The following function aims to replicate the way np.convolve is computing the output values: def mov_avg(x, w): for m in range(len(x)-(w-1)): yield sum(np.ones(w) * x[m:m+w]) / w Which, for the same example above would also yield: list(mov_avg(x, 2)) # [4.0, 5.5, 9.0, 6.0, 1.5, 3.0, 3.0, 0.5, 1.0] So what is being done at each step is to take the inner product between the array of ones and the current window. In this case the multiplication by np.ones(w) is superfluous given that we are directly taking the sum of the sequence. Below is an example of how the first outputs are computed so that it is a little clearer. Lets suppose we want a window of w=4: [1,1,1,1] [5,3,8,10,2,1,5,1,0,2] = (1*5 + 1*3 + 1*8 + 1*10) / w = 6.5 And the following output would be computed as: [1,1,1,1] [5,3,8,10,2,1,5,1,0,2] = (1*3 + 1*8 + 1*10 + 1*2) / w = 5.75 And so on, returning a moving average of the sequence once all overlaps have been performed.


Instruction:
How to make good reproducible pandas examples

Having spent a decent amount of time watching both the r and pandas tags on SO, the impression that I get is that pandas questions are less likely to contain reproducible data. This is something that the R community has been pretty good about encouraging, and thanks to guides like this, newcomers are able to get some help on putting together these examples. People who are able to read these guides and come back with reproducible data will often have much better luck getting answers to their questions. How can we create good reproducible examples for pandas questions? Simple dataframes can be put together, e.g.: import pandas as pd df = pd.DataFrame({'user': ['Bob', 'Jane', 'Alice'], 'income': [40000, 50000, 42000]}) But many example datasets need more complicated structure, e.g.: datetime indices or data Multiple categorical variables (is there an equivalent to R's expand.grid() function, which produces all possible combinations of some given variables?) MultiIndex data For datasets that are hard to mock up using a few lines of code, is there an equivalent to R's dput() that allows you to generate copy-pasteable code to regenerate your datastructure?

Output: Note: Most of the ideas here are pretty generic for Stack Overflow, indeed questions in general. See Minimal, Reproducible Example or Short, Self Contained, Correct Example. Disclaimer: Writing a good question is hard. The Good: Do include a small example DataFrame, either as runnable code: In [1]: df = pd.DataFrame([[1, 2], [1, 3], [4, 6]], columns=['A', 'B']) or make it "copy and pasteable" using pd.read_clipboard(sep=r'\s\s+'). In [2]: df Out[2]: A B 0 1 2 1 1 3 2 4 6 Test it yourself to make sure it works and reproduces the issue. You can format the text for Stack Overflow by highlighting and using Ctrl+K (or prepend four spaces to each line), or place three backticks (```) above and below your code with your code unindented. I really do mean small. The vast majority of example DataFrames could be fewer than 6 rows and 6 columns,[citation needed] and I bet I can do it in 5x3. Can you reproduce the error with df = df.head()[relevant_columns]? If not, fiddle around to see if you can make up a small DataFrame which exhibits the issue you are facing. But every rule has an exception, the obvious one being for performance issues (in which case definitely use %timeit and possibly %prun to profile your code), where you should generate: df = pd.DataFrame(np.random.randn(100000000, 10)) Consider using np.random.seed so we have the exact same frame. Having said that, "make this code fast for me" is not strictly on topic for the site. For getting runnable code, df.to_dict is often useful, with the different orient options for different cases. In the example above, I could have grabbed the columns and values from df.to_dict('split'). Write out the outcome you desire (similarly to above) In [3]: iwantthis Out[3]: A B 0 1 5 1 4 6 Explain where the numbers come from: The 5 is the sum of the B column for the rows where A is 1. Do show the code you've tried: In [4]: df.groupby('A').sum() Out[4]: B A 1 5 4 6 But say what's incorrect: The A column is in the index rather than a column. Do show you've done some research (search the documentation, search Stack Overflow), and give a summary: The docstring for sum simply states "Compute sum of group values" The groupby documentation doesn't give any examples for this. Aside: the answer here is to use df.groupby('A', as_index=False).sum(). If it's relevant that you have Timestamp columns, e.g. you're resampling or something, then be explicit and apply pd.to_datetime to them for good measure. df['date'] = pd.to_datetime(df['date']) # this column ought to be date. Sometimes this is the issue itself: they were strings. The Bad: Don't include a MultiIndex, which we can't copy and paste (see above). This is kind of a grievance with Pandas' default display, but nonetheless annoying: In [11]: df Out[11]: C A B 1 2 3 2 6 The correct way is to include an ordinary DataFrame with a set_index call: In [12]: df = pd.DataFrame([[1, 2, 3], [1, 2, 6]], columns=['A', 'B', 'C']) In [13]: df = df.set_index(['A', 'B']) In [14]: df Out[14]: C A B 1 2 3 2 6 Do provide insight to what it is when giving the outcome you want: B A 1 1 5 0 Be specific about how you got the numbers (what are they)... double check they're correct. If your code throws an error, do include the entire stack trace. This can be edited out later if it's too noisy. Show the line number and the corresponding line of your code which it's raising against. Pandas 2.0 introduced a number of changes, and Pandas 1.0 before that, so if you're getting unexpected output, include the version: pd.__version__ On that note, you might also want to include the version of Python, your OS, and any other libraries. You could use pd.show_versions() or the session_info package (which shows loaded libraries and Jupyter/IPython environment). The Ugly: Don't link to a CSV file we don't have access to (and ideally don't link to an external source at all). df = pd.read_csv('my_secret_file.csv') # ideally with lots of parsing options Most data is proprietary, we get that. Make up similar data and see if you can reproduce the problem (something small). Don't explain the situation vaguely in words, like you have a DataFrame which is "large", mention some of the column names in passing (be sure not to mention their dtypes). Try and go into lots of detail about something which is completely meaningless without seeing the actual context. Presumably no one is even going to read to the end of this paragraph. Essays are bad; it's easier with small examples. Don't include 10+ (100+??) lines of data munging before getting to your actual question. Don't include long strings, large numbers, floats with lots of decimals, or other lengthy scalars if you can avoid it. Please, we see enough of this in our day jobs. We want to help, but not like this.... Cut the intro, and just show the relevant data (or a small, simplified version of it) in the step which is causing you trouble.


Instruction:
Convert to binary and keep leading zeros

I'm trying to convert an integer to binary using the bin() function in Python. However, it always removes the leading zeros, which I actually need, such that the result is always 8-bit: Example: bin(1) -> 0b1 # What I would like: bin(1) -> 0b00000001 Is there a way of doing this?

Output: Use the format() function: >>> format(14, '#010b') '0b00001110' The format() function simply formats the input following the Format Specification mini language. The # makes the format include the 0b prefix, and the 010 size formats the output to fit in 10 characters width, with 0 padding; 2 characters for the 0b prefix, the other 8 for the binary digits. This is the most compact and direct option. If you are putting the result in a larger string, use an formatted string literal (3.6+) or use str.format() and put the second argument for the format() function after the colon of the placeholder {:..}: >>> value = 14 >>> f'The produced output, in binary, is: {value:#010b}' 'The produced output, in binary, is: 0b00001110' >>> 'The produced output, in binary, is: {:#010b}'.format(value) 'The produced output, in binary, is: 0b00001110' As it happens, even for just formatting a single value (so without putting the result in a larger string), using a formatted string literal is faster than using format(): >>> import timeit >>> timeit.timeit("f_(v, '#010b')", "v = 14; f_ = format") # use a local for performance 0.40298633499332936 >>> timeit.timeit("f'{v:#010b}'", "v = 14") 0.2850222919951193 But I'd use that only if performance in a tight loop matters, as format(...) communicates the intent better. If you did not want the 0b prefix, simply drop the # and adjust the length of the field: >>> format(14, '08b') '00001110'


Instruction:
How do I combine two dataframes?

I have a initial dataframe D. I extract two data frames from it like this: A = D[D.label == k] B = D[D.label != k] I want to combine A and B into one DataFrame. The order of the data is not important. However, when we sample A and B from D, they retain their indexes from D.

Output: Updated in 2025 df_merged = pandas.concat([df1, df2], ignore_index=True, sort=False) ** OLD ANSWER** DEPRECATED: DataFrame.append and Series.append were deprecated in v1.4.0. Use append: df_merged = df1.append(df2, ignore_index=True) And to keep their indexes, set ignore_index=False.


Instruction:
How to erase the file contents of text file in Python?

I have text file which I want to erase in Python. How do I do that?

Output: In Python: open('file.txt', 'w').close() Or alternatively, if you have already an opened file: f = open('file.txt', 'r+') f.truncate(0) # need '0' when using r+


Instruction:
What does .contiguous() do in PyTorch?

What does x.contiguous() do for a tensor x?

Output: There are a few operations on Tensors in PyTorch that do not change the contents of a tensor, but change the way the data is organized. These operations include: narrow(), view(), expand() and transpose() For example: when you call transpose(), PyTorch doesn't generate a new tensor with a new layout, it just modifies meta information in the Tensor object so that the offset and stride describe the desired new shape. In this example, the transposed tensor and original tensor share the same memory: x = torch.randn(3,2) y = torch.transpose(x, 0, 1) x[0, 0] = 42 print(y[0,0]) # prints 42 This is where the concept of contiguous comes in. In the example above, x is contiguous but y is not because its memory layout is different to that of a tensor of same shape made from scratch. Note that the word "contiguous" is a bit misleading because it's not that the content of the tensor is spread out around disconnected blocks of memory. Here bytes are still allocated in one block of memory but the order of the elements is different! When you call contiguous(), it actually makes a copy of the tensor such that the order of its elements in memory is the same as if it had been created from scratch with the same data. Normally you don't need to worry about this. You're generally safe to assume everything will work, and wait until you get a RuntimeError: input is not contiguous where PyTorch expects a contiguous tensor to add a call to contiguous().


Instruction:
Convert a namedtuple into a dictionary

I have a named tuple class in python class Town(collections.namedtuple('Town', [ 'name', 'population', 'coordinates', 'population', 'capital', 'state_bird'])): # ... I'd like to convert Town instances into dictionaries. I don't want it to be rigidly tied to the names or number of the fields in a Town. Is there a way to write it such that I could add more fields, or pass an entirely different named tuple in and get a dictionary. I can not alter the original class definition as its in someone else's code. So I need to take an instance of a Town and convert it to a dictionary.

Output: TL;DR: there's a method _asdict provided for this. Here is a demonstration of the usage: >>> from collections import namedtuple >>> fields = ['name', 'population', 'coordinates', 'capital', 'state_bird'] >>> Town = namedtuple('Town', fields) >>> funkytown = Town('funky', 300, 'somewhere', 'lipps', 'chicken') >>> funkytown._asdict() {'name': 'funky', 'population': 300, 'coordinates': 'somewhere', 'capital': 'lipps', 'state_bird': 'chicken'} This is a documented method of namedtuples, i.e. unlike the usual convention in python the leading underscore on the method name isn't there to discourage use. Along with the other methods added to namedtuples, _make, _replace, _source, _fields, it has the underscore only to try and prevent conflicts with possible field names. Note: For some 2.7.5 < python version < 3.5.0 code out in the wild, you might see this version: >>> vars(funkytown) OrderedDict([('name', 'funky'), ('population', 300), ('coordinates', 'somewhere'), ('capital', 'lipps'), ('state_bird', 'chicken')]) For a while the documentation had mentioned that _asdict was obsolete (see here), and suggested to use the built-in method vars. That advice is now outdated; in order to fix a bug related to subclassing, the __dict__ property which was present on namedtuples has again been removed by this commit.


Instruction:
How to merge a transparent png image with another image using PIL

I have a transparent png image foo.png and I've opened another image with: im = Image.open("foo2.png") Now what I need is to merge foo.png with foo2.png. (foo.png contains some text and I want to print that text on foo2.png)

Output: from PIL import Image background = Image.open("test1.png") foreground = Image.open("test2.png") background.paste(foreground, (0, 0), foreground) background.show() First parameter to .paste() is the image to paste. Second are coordinates, and the secret sauce is the third parameter. It indicates a mask that will be used to paste the image. If you pass a image with transparency, then the alpha channel is used as mask. Check the docs.


Instruction:
Error message "error: command 'gcc' failed with exit status 1" while installing eventlet

I wanted to install Eventlet on my system in order to have "Herd" for software deployment, but the terminal is showing a GCC error (as root): cd ~ easy_install -U eventlet Output: Searching for eventlet Reading http://pypi.python.org/simple/eventlet/ Reading http://wiki.secondlife.com/wiki/Eventlet Reading http://eventlet.net Best match: eventlet 0.9.16 Processing eventlet-0.9.16-py2.7.egg eventlet 0.9.16 is already the active version in easy-install.pth Using /usr/local/lib/python2.7/dist-packages/eventlet-0.9.16-py2.7.egg Processing dependencies for eventlet Searching for greenlet>=0.3 Reading http://pypi.python.org/simple/greenlet/ Reading https://github.com/python-greenlet/greenlet Reading http://bitbucket.org/ambroff/greenlet Best match: greenlet 0.3.4 Downloading http://pypi.python.org/packages/source/g/greenlet/greenlet- 0.3.4.zip#md5=530a69acebbb0d66eb5abd83523d8272 Processing greenlet-0.3.4.zip Writing /tmp/easy_install-_aeHYm/greenlet-0.3.4/setup.cfg Running greenlet-0.3.4/setup.py -q bdist_egg --dist-dir /tmp/easy_install-_aeHYm/greenlet-0.3.4/egg-dist-tmp-t9_gbW In file included from greenlet.c:5:0: greenlet.h:8:20: fatal error: Python.h: No such file or directory compilation terminated. error: Setup script exited with error: command 'gcc' failed with exit status 1` Why can't Python.h be found?

Output: Your install is failing because you don't have the Python development headers installed. First, update the packages with sudo apt update. You can do this through APT on Ubuntu/Debian with: sudo apt-get install python-dev For Python 3, use: sudo apt-get install python3-dev For eventlet you might also need the libevent libraries installed so if you get an error talking about that you can install libevent with: sudo apt-get install libevent-dev


Instruction:
tqdm in Jupyter Notebook prints new progress bars repeatedly

I am using tqdm to print progress in a script I'm running in a Jupyter notebook. I am printing all messages to the console via tqdm.write(). However, this still gives me a skewed output like so: That is, each time a new line has to be printed, a new progress bar is printed on the next line. This does not happen when I run the script via terminal. How can I solve this?

Output: Try using tqdm.notebook.tqdm instead of tqdm, as outlined here. This could be as simple as changing your import to: from tqdm.notebook import tqdm EDIT: After testing, it seems that tqdm actually works fine in 'text mode' in Jupyter notebook. It's hard to tell because you haven't provided a minimal example, but it looks like your problem is caused by a print statement in each iteration. The print statement is outputting a number (~0.89) in between each status bar update, which is messing up the output. Try removing the print statement.


Instruction:
collapse cell in jupyter notebook

I am using ipython Jupyter notebook. Let's say I defined a function that occupies a lot of space on my screen. Is there a way to collapse the cell? I want the function to remain executed and callable, yet I want to hide / collapse the cell in order to better visualize the notebook. How can I do this?

Output: UPDATE: The newer jupyter-lab is a more modern and feature-rich interface which supports cell folding by default. See @intsco's answer below UPDATE 2 Since jupyter-lab now also supports extensions, you can extend the built-in cell-folding functionality with the Collapsible_Headings extension. Original answer: The jupyter contrib nbextensions Python package contains a code-folding extension that can be enabled within the notebook. Follow the link (Github) for documentation. To install using command line: pip install jupyter_contrib_nbextensions jupyter contrib nbextension install --user To make life easier in managing them, I'd also recommend the jupyter nbextensions configurator package. This provides an extra tab in your Notebook interface from where you can easily (de)activate all installed extensions. Installation: pip install jupyter_nbextensions_configurator jupyter nbextensions_configurator enable --user


Instruction:
Python memory leaks

I have a long-running script which, if let to run long enough, will consume all the memory on my system. Without going into details about the script, I have two questions: Are there any "Best Practices" to follow, which will help prevent leaks from occurring? What techniques are there to debug memory leaks in Python?

Output: Have a look at this article: Tracing python memory leaks Also, note that the garbage collection module actually can have debug flags set. Look at the set_debug function. Additionally, look at this code by Gnibbler for determining the types of objects that have been created after a call.


Instruction:
How to get the seconds since epoch from the time + date output of gmtime()?

How do you do reverse gmtime(), where you put the time + date and get the number of seconds? I have strings like 'Jul 9, 2009 @ 20:02:58 UTC', and I want to get back the number of seconds between the epoch and July 9, 2009. I have tried time.strftime but I don't know how to use it properly, or if it is the correct command to use.

Output: If you got here because a search engine told you this is how to get the Unix timestamp, stop reading this answer. Scroll up one. If you want to reverse time.gmtime(), you want calendar.timegm(). >>> calendar.timegm(time.gmtime()) 1293581619.0 You can turn your string into a time tuple with time.strptime(), which returns a time tuple that you can pass to calendar.timegm(): >>> import calendar >>> import time >>> calendar.timegm(time.strptime('Jul 9, 2009 @ 20:02:58 UTC', '%b %d, %Y @ %H:%M:%S UTC')) 1247169778 More information about calendar module here


Instruction:
Make a new list containing every Nth item in the original list

Say we have a list of integers from 0 to 1000: [0, 1, 2, 3, ..., 997, 998, 999] How do I get a new list containing the first and every subsequent 10th item? [0, 10, 20, 30, ..., 990] I can do this using a for loop, but is there a neater way, perhaps even in one line of code?

Output: >>> xs = list(range(165)) >>> xs[0::10] [0, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 110, 120, 130, 140, 150, 160] Note that this is around 100 times faster than looping and checking a modulus for each element: $ python -m timeit -s "xs = list(range(1000))" "[x for i, x in enumerate(xs) if i % 10 == 0]" 500 loops, best of 5: 476 usec per loop $ python -m timeit -s "xs = list(range(1000))" "xs[0::10]" 100000 loops, best of 5: 3.32 usec per loop


Instruction:
Ignoring NaNs with str.contains

I want to find rows that contain a string, like so: DF[DF.col.str.contains("foo")] However, this fails because some elements are NaN: ValueError: cannot index with vector containing NA / NaN values So I resort to the obfuscated DF[DF.col.notnull()][DF.col.dropna().str.contains("foo")] Is there a better way?

Output: There's a flag for that: In [11]: df = pd.DataFrame([["foo1"], ["foo2"], ["bar"], [np.nan]], columns=['a']) In [12]: df.a.str.contains("foo") Out[12]: 0 True 1 True 2 False 3 NaN Name: a, dtype: object In [13]: df.a.str.contains("foo", na=False) Out[13]: 0 True 1 True 2 False 3 False Name: a, dtype: bool See the str.replace docs: na : default NaN, fill value for missing values. So you can do the following: In [21]: df.loc[df.a.str.contains("foo", na=False)] Out[21]: a 0 foo1 1 foo2


Instruction:
Asynchronous requests with Python's Requests

I tried the sample provided within the documentation of the Requests library for Python. With async.map(rs), I get the response codes, but I want to get the content of each page requested. This, for example, does not work: out = async.map(rs) print out[0].content

Output: Note The below answer is not applicable to Requests v0.13.0 or later. The asynchronous functionality was moved to GRequests after this question was written. However, you could just replace requests with grequests below and it should work. I've left this answer as is to reflect the original question which was about using requests < v0.13.0. To do multiple tasks with async.map asynchronously you have to: Define a function for what you want to do with each object (your task) Add that function as an event hook in your request Call async.map on a list of all the requests / actions Example: from requests import async # If using requests > v0.13.0, use # from grequests import async urls = [ 'http://python-requests.org', 'http://httpbin.org', 'http://python-guide.org', 'http://kennethreitz.com' ] # A simple task to do to each response object def do_something(response): print response.url # A list to hold our things to do via async async_list = [] for u in urls: # The "hooks = {..." part is where you define what you want to do # # Note the lack of parentheses following do_something, this is # because the response will be used as the first argument automatically action_item = async.get(u, hooks = {'response' : do_something}) # Add the task to our list of things to do via async async_list.append(action_item) # Do our list of things to do via async async.map(async_list)


Instruction:
How is set() implemented?

I've seen people say that set objects in python have O(1) membership-checking. How are they implemented internally to allow this? What sort of data structure does it use? What other implications does that implementation have? Every answer here was really enlightening, but I can only accept one, so I'll go with the closest answer to my original question. Thanks all for the info!

Output: According to this thread: Indeed, CPython's sets are implemented as something like dictionaries with dummy values (the keys being the members of the set), with some optimization(s) that exploit this lack of values So basically a set uses a hashtable as its underlying data structure. This explains the O(1) membership checking, since looking up an item in a hashtable is an O(1) operation, on average. If you are so inclined you can even browse the CPython source code for set which, according to Achim Domma, was originally mostly a cut-and-paste from the dict implementation. Note: Nowadays, set and dict's implementations have diverged significantly, so the precise behaviors (e.g. arbitrary order vs. insertion order) and performance in various use cases differs; they're still implemented in terms of hashtables, so average case lookup and insertion remains O(1), but set is no longer just "dict, but with dummy/omitted values".


Instruction:
Why does Python's hash of infinity have the digits of π?

The hash of infinity in Python has digits matching pi: >>> inf = float('inf') >>> hash(inf) 314159 >>> int(math.pi*1e5) 314159 Is that just a coincidence or is it intentional?

Output: _PyHASH_INF is defined as a constant equal to 314159. I can't find any discussion about this, or comments giving a reason. I think it was chosen more or less arbitrarily. I imagine that as long as they don't use the same meaningful value for other hashes, it shouldn't matter.


Instruction:
How to remove all characters after a specific character in python?

I have a string. How do I remove all text after a certain character? (In this case ...) The text after will ... change so I that's why I want to remove all characters after a certain one.

Output: Split on your separator at most once, and take the first piece: sep = '...' stripped = text.split(sep, 1)[0] You didn't say what should happen if the separator isn't present. Both this and Alex's solution will return the entire string in that case.


Instruction:
How to create key or append an element to key?

I have an empty dictionary. Name: dict_x It is to have keys of which values are lists. From a separate iteration, I obtain a key (ex: key_123), and an item (a tuple) to place in the list of dict_x's value key_123. If this key already exists, I want to append this item. If this key does not exist, I want to create it with an empty list and then append to it or just create it with a tuple in it. In future when again this key comes up, since it exists, I want the value to be appended again. My code consists of this: Get key and value. See if NOT key exists in dict_x. and if not create it: dict_x[key] == [] Afterwards: dict_x[key].append(value) Is this the way to do it? Shall I try to use try/except blocks?

Output: Use dict.setdefault(): dict.setdefault(key,[]).append(value) help(dict.setdefault): setdefault(...) D.setdefault(k[,d]) -> D.get(k,d), also set D[k]=d if k not in D


Instruction:
How can I break up this long line in Python?

How would you go about formatting a long line such as this? I'd like to get it to no more than 80 characters wide: logger.info("Skipping {0} because its thumbnail was already in our system as {1}.".format(line[indexes['url']], video.title)) Is this my best option? url = "Skipping {0} because its thumbnail was already in our system as {1}." logger.info(url.format(line[indexes['url']], video.title))

Output: That's a start. It's not a bad practice to define your longer strings outside of the code that uses them. It's a way to separate data and behavior. Your first option is to join string literals together implicitly by making them adjacent to one another: ("This is the first line of my text, " "which will be joined to a second.") Or with line ending continuations, which is a little more fragile, as this works: "This is the first line of my text, " \ "which will be joined to a second." But this doesn't: "This is the first line of my text, " \ "which will be joined to a second." See the difference? No? Well you won't when it's your code either. (There's a space after \ in the second example.) The downside to implicit joining is that it only works with string literals, not with strings taken from variables, so things can get a little more hairy when you refactor. Also, you can only interpolate formatting on the combined string as a whole. Alternatively, you can join explicitly using the concatenation operator (+): ("This is the first line of my text, " + "which will be joined to a second.") Explicit is better than implicit, as the zen of python says, but this creates three strings instead of one, and uses twice as much memory: there are the two you have written, plus one which is the two of them joined together, so you have to know when to ignore the zen. The upside is you can apply formatting to any of the substrings separately on each line, or to the whole lot from outside the parentheses. Finally, you can use triple-quoted strings: """This is the first line of my text which will be joined to a second.""" This is often my favorite, though its behavior is slightly different as the newline and any leading whitespace on subsequent lines will show up in your final string. You can eliminate the newline with an escaping backslash. """This is the first line of my text \ which will be joined to a second.""" This has the same problem as the same technique above, in that correct code only differs from incorrect code by invisible whitespace. Which one is "best" depends on your particular situation, but the answer is not simply aesthetic, but one of subtly different behaviors.


Instruction:
How to change the name of a Django app?

I have changed the name of an app in Django by renaming its folder, imports and all its references (templates/indexes). But now I get this error when I try to run python manage.py runserver Error: Could not import settings 'nameofmynewapp.settings' (Is it on sys.path?): No module named settings How can I debug and solve this error? Any clues?

Output: Follow these steps to change an app's name in Django: Rename the folder which is in your project root Change any references to your app in their dependencies, i.e. the app's views.py, urls.py , manage.py , and settings.py files. Edit the database table django_content_type with the following command: UPDATE django_content_type SET app_label='<NewAppName>' WHERE app_label='<OldAppName>' Also, if you have models, you will have to rename the model tables. For postgres, use ALTER TABLE <oldAppName>_modelName RENAME TO <newAppName>_modelName. For mysql too, I think it is the same (as mentioned by @null_radix). (For Django >= 1.7) Update the django_migrations table to avoid having your previous migrations re-run: UPDATE django_migrations SET app='<NewAppName>' WHERE app='<OldAppName>'. Note: there is some debate (in comments) if this step is required for Django 1.8+; If someone knows for sure please update here. If your models.py 's Meta Class has app_name listed, make sure to rename that too (mentioned by @will). If you've namespaced your static or templates folders inside your app, you'll also need to rename those. For example, rename old_app/static/old_app to new_app/static/new_app. For renaming django models, you'll need to change django_content_type.name entry in DB. For postgreSQL, use UPDATE django_content_type SET name='<newModelName>' where name='<oldModelName>' AND app_label='<OldAppName>' Update 16Jul2021: Also, the __pycache__/ folder inside the app must be removed, otherwise you get EOFError: marshal data too short when trying to run the server. Mentioned by @Serhii Kushchenko Update the apps.py file in the app you're renaming to use the new name. Change the "<old_app_name>Config" class name to "<new_app_name>Config" and within this class change name = '<old_app_name>' to name = '<new_app_name>' Meta point (If using virtualenv): Worth noting, if you are renaming the directory that contains your virtualenv, there will likely be several files in your env that contain an absolute path and will also need to be updated. If you are getting errors such as ImportError: No module named ... this might be the culprit. (thanks to @danyamachine for providing this). Other references: you might also want to refer to the below links for a more complete picture: Renaming an app with Django and South How do I migrate a model out of one django app and into a new one? How to change the name of a Django app? Backwards migration with Django South Easiest way to rename a model using Django/South? Python code (thanks to A.Raouf) to automate the above steps (Untested code. You have been warned!) Python code (thanks to rafaponieman) to automate the above steps (Untested code. You have been warned!)


Instruction:
How to solve "error: Microsoft Visual C++ 14.0 or greater is required" when installing Python packages?

I'm trying to install a package on Python, but Python is throwing an error on installing packages. I'm getting an error every time I tried to install pip install google-search-api. Here is the error how can I successfully install it? error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/ I already updated that and have the latest version of 14.27 but the problem is throwing the same error.

Output: Go to this link and download Microsoft C++ Build Tools: https://visualstudio.microsoft.com/visual-cpp-build-tools/ Open the installer, then follow the steps. You might have something like this, just download it or resume. If updating above doesn't work then you need to configure or make some updates here. You can make some updates here too by clicking "Modify". Check that and download what you need there or you might find that you just need to update Microsoft Visual C++ as stated on the error, but I also suggest updating everything there because you might still need it on your future programs. I think those with the C++ as I've done that before and had a similar problem just like that when installing a python package for creating WorldCloud visualization. UPDATE: December 28, 2020 You can also follow these steps here: Select: Workloads → Desktop development with C++ Then for Individual Components, select only: Windows 10 SDK C++ x64/x86 build tools You can also achieve the same automatically using the following command: vs_buildtools.exe --norestart --passive --downloadThenInstall --includeRecommended --add Microsoft.VisualStudio.Workload.NativeDesktop --add Microsoft.VisualStudio.Workload.VCTools --add Microsoft.VisualStudio.Workload.MSBuildTools Reference: https://www.scivision.dev/python-windows-visual-c-14-required


Instruction:
Converting a list to a set changes element order

Recently I noticed that when I am converting a list to set the order of elements is changed and is sorted by character. Consider this example: x=[1,2,20,6,210] print(x) # [1, 2, 20, 6, 210] # the order is same as initial order set(x) # set([1, 2, 20, 210, 6]) # in the set(x) output order is sorted My questions are - Why is this happening? How can I do set operations (especially set difference) without losing the initial order?

Output: A set is an unordered data structure, so it does not preserve the insertion order. CPython's set enumeration order depends on last bits of the element's hash and whether there have been key collisions during insertion. This depends on your requirements. If you have an normal list, and want to remove some set of elements while preserving the order of the list, you can do this with a list comprehension: >>> a = [1, 2, 20, 6, 210] >>> b = set([6, 20, 1]) >>> [x for x in a if x not in b] [2, 210] If you need a data structure that supports both fast membership tests and preservation of insertion order, you can use the keys of a Python dictionary, which starting from Python 3.7 is guaranteed to preserve the insertion order: >>> a = dict.fromkeys([1, 2, 20, 6, 210]) >>> b = dict.fromkeys([6, 20, 1]) >>> dict.fromkeys(x for x in a if x not in b) {2: None, 210: None} b doesn't really need to be ordered here – you could use a set as well. Note that a.keys() - b.keys() returns the set difference as a set, so it won't preserve the insertion order. In older versions of Python, you can use collections.OrderedDict instead: >>> a = collections.OrderedDict.fromkeys([1, 2, 20, 6, 210]) >>> b = collections.OrderedDict.fromkeys([6, 20, 1]) >>> collections.OrderedDict.fromkeys(x for x in a if x not in b) OrderedDict([(2, None), (210, None)])


Instruction:
How can I access a dictionary element in a Django template?

I would like to print out the number of votes that each choice got. I have this code in a template: {% for choice in choices %} {{choice.choice}} - {{votes[choice.id]}} <br /> {% endfor %} votes is just a dictionary while choices is a model object. It raises an exception with this message: "Could not parse the remainder"

Output: To echo / extend upon Jeff's comment, I think you should simply aim for a property in your Choice class that calculates the number of votes associated with that object: class Choice(models.Model): text = models.CharField(max_length=200) def calculateVotes(self): return Vote.objects.filter(choice=self).count() votes = property(calculateVotes) And then in your template, you can do: {% for choice in choices %} {{choice.choice}} - {{choice.votes}} <br /> {% endfor %} The template tag, is, in my humble opinion, a bit overkill for this solution, but it's not a terrible solution either. The goal of templates in Django is to insulate you from code in your templates and vice versa. I'd try the above method and see what SQL the ORM generates as I'm not sure off the top of my head if it will precache the properties and just create a subselect for the property or if it will iteratively / on-demand run the query to calculate vote count. But if it generates atrocious queries, you could always populate the property in your view with data you've collected yourself.


Instruction:
Generate random numbers with a given (numerical) distribution

I have a file with some probabilities for different values e.g.: 1 0.1 2 0.05 3 0.05 4 0.2 5 0.4 6 0.2 I would like to generate random numbers using this distribution. Does an existing module that handles this exist? It's fairly simple to code on your own (build the cumulative density function, generate a random value [0,1] and pick the corresponding value) but it seems like this should be a common problem and probably someone has created a function/module for it. I need this because I want to generate a list of birthdays (which do not follow any distribution in the standard random module).

Output: scipy.stats.rv_discrete might be what you want. You can supply your probabilities via the values parameter. You can then use the rvs() method of the distribution object to generate random numbers. As pointed out by Eugene Pakhomov in the comments, you can also pass a p keyword parameter to numpy.random.choice(), e.g. numpy.random.choice(numpy.arange(1, 7), p=[0.1, 0.05, 0.05, 0.2, 0.4, 0.2]) If you are using Python 3.6 or above, you can use random.choices() from the standard library – see the answer by Mark Dickinson.


Instruction:
AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?

Earlier I installed some packages like Matplotlib, NumPy, pip (version 23.3.1), wheel (version 0.41.2), etc., and did some programming with those. I used the command C:\Users\UserName>pip list to find the list of packages that I have installed, and I am using Python 3.12.0 (by employing code C:\Users\UserName>py -V). I need to use pyspedas to analyse some data. I am following the instruction that that I received from site to install the package, with a variation (I am not sure whether it matters or not: I am using py, instead of python). The commands that I use, in the order, are: py -m venv pyspedas .\pyspedas\Scripts\activate pip install pyspedas After the last step, I am getting the following output: Collecting pyspedas Using cached pyspedas-1.4.47-py3-none-any.whl.metadata (14 kB) Collecting numpy>=1.19.5 (from pyspedas) Using cached numpy-1.26.1-cp312-cp312-win_amd64.whl.metadata (61 kB) Collecting requests (from pyspedas) Using cached requests-2.31.0-py3-none-any.whl.metadata (4.6 kB) Collecting geopack>=1.0.10 (from pyspedas) Using cached geopack-1.0.10-py3-none-any.whl (114 kB) Collecting cdflib<1.0.0 (from pyspedas) Using cached cdflib-0.4.9-py3-none-any.whl (72 kB) Collecting cdasws>=1.7.24 (from pyspedas) Using cached cdasws-1.7.43.tar.gz (21 kB) Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done Collecting netCDF4>=1.6.2 (from pyspedas) Using cached netCDF4-1.6.5-cp312-cp312-win_amd64.whl.metadata (1.8 kB) Collecting pywavelets (from pyspedas) Using cached PyWavelets-1.4.1.tar.gz (4.6 MB) Installing build dependencies ... done Getting requirements to build wheel ... error error: subprocess-exited-with-error × Getting requirements to build wheel did not run successfully. │ exit code: 1 ╰─> [33 lines of output] Traceback (most recent call last): File "C:\Users\UserName\pyspedas\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 353, in <module> main() File "C:\Users\UserName\pyspedas\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 335, in main json_out['return_val'] = hook(**hook_input['kwargs']) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\UserName\pyspedas\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 112, in get_requires_for_build_wheel backend = _build_backend() ^^^^^^^^^^^^^^^^ File "C:\Users\UserName\pyspedas\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 77, in _build_backend obj = import_module(mod_path) ^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\UserName\AppData\Local\Programs\Python\Python312\Lib\importlib\__init__.py", line 90, in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "<frozen importlib._bootstrap>", line 1381, in _gcd_import File "<frozen importlib._bootstrap>", line 1354, in _find_and_load File "<frozen importlib._bootstrap>", line 1304, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed File "<frozen importlib._bootstrap>", line 1381, in _gcd_import File "<frozen importlib._bootstrap>", line 1354, in _find_and_load File "<frozen importlib._bootstrap>", line 1325, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 929, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 994, in exec_module File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed File "C:\Users\UserName\AppData\Local\Temp\pip-build-env-_lgbq70y\overlay\Lib\site-packages\setuptools\__init__.py", line 16, in <module> import setuptools.version File "C:\Users\UserName\AppData\Local\Temp\pip-build-env-_lgbq70y\overlay\Lib\site-packages\setuptools\version.py", line 1, in <module> import pkg_resources File "C:\Users\UserName\AppData\Local\Temp\pip-build-env-_lgbq70y\overlay\Lib\site-packages\pkg_resources\__init__.py", line 2191, in <module> register_finder(pkgutil.ImpImporter, find_on_path) ^^^^^^^^^^^^^^^^^^^ AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'? [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. error: subprocess-exited-with-error × Getting requirements to build wheel did not run successfully. │ exit code: 1 ╰─> See above for output. note: This error originates from a subprocess, and is likely not a problem with pip. After little bit of googling, I came to know that this issues was reported at multiple places, but none for this package. I did install wheel in the new environment as mentioned in the answer here, but the problem still persists. Instead of setting up a virtual environment, I simply executed the command py -m pip install pyspedas. But I am still getting the error. What I could gather is that the program has an issue with Collecting pywavelets (from pyspedas) Using cached PyWavelets-1.4.1.tar.gz (4.6 MB) Installing build dependencies ... done I am using IDLE in Windows 11.

Output: Due to the removal of the long-deprecated pkgutil.ImpImporter class, the pip command may not work for Python 3.12. You just have to manually install pip for Python 3.12 python -m ensurepip --upgrade python -m pip install --upgrade setuptools python -m pip install <module> In your virtual environment: pip install --upgrade setuptools Python comes with an ensurepip, which can install pip in a Python environment. https://pip.pypa.io/en/stable/installation/ On Linux/macOS terminal: python -m ensurepip --upgrade On Windows: py -m ensurepip --upgrade also, make sure to upgrade pip: py -m pip install --upgrade pip To install numpy on Python 3.12, you must use numpy version 1.26.4 pip install numpy==1.26.4 https://github.com/numpy/numpy/issues/23808#issuecomment-1722440746 for Ubuntu sudo apt install python3.12-dev or python3.12 -m pip install --upgrade setuptools


Instruction:
How to change default Anaconda python environment

I've installed Anaconda and created two extra environments: py3k (which holds Python 3.3) and py34 (which holds Python 3.4). Besides those, I have a default environment named 'root' which the Anaconda installer created by default and which holds Python 2.7. This last one is the default, whenever I launch 'ipython' from the terminal it gives me version 2.7. In order to work with Python 3.4, I need to issue the commands (in the shell) source activate py34 ipython which change the default environment to Python 3.4. This works fine, but it's annoying since most of the time I work on Python 3.4, instead of Python 2.7 (which I hold for teaching purposes, it's a rather long story). Anyway, I'll like to know how to change the default environment to Python 3.4, bearing in mind that I don't want to reinstall everything from scratch.

Output: If you just want to temporarily change to another environment, use source activate environment-name ETA: This may be deprecated. I believe the current correct command is: source conda activate environment-name (you can create environment-name with conda create) To change permanently, there is no method except creating a startup script that runs the above code. Typically it's best to just create new environments. However, if you really want to change the Python version in the default environment, you can do so as follows: First, make sure you have the latest version of conda by running conda update conda Then run conda install python=3.5 This will attempt to update all your packages in your root environment to Python 3 versions. If it is not possible (e.g., because some package is not built for Python 3.5), it will give you an error message indicating which package(s) caused the issue. If you installed packages with pip, you'll have to reinstall them.


Instruction:
Python Mocking a function from an imported module

I want to understand how to @patch a function from an imported module. This is where I am so far. app/mocking.py: from app.my_module import get_user_name def test_method(): return get_user_name() if __name__ == "__main__": print "Starting Program..." test_method() app/my_module/__init__.py: def get_user_name(): return "Unmocked User" test/mock-test.py: import unittest from app.mocking import test_method def mock_get_user(): return "Mocked This Silly" @patch('app.my_module.get_user_name') class MockingTestTestCase(unittest.TestCase): def test_mock_stubs(self, mock_method): mock_method.return_value = 'Mocked This Silly') ret = test_method() self.assertEqual(ret, 'Mocked This Silly') if __name__ == '__main__': unittest.main() This does not work as I would expect. The "patched" module simply returns the unmocked value of get_user_name. How do I mock methods from other packages that I am importing into a namespace under test?

Output: When you are using the patch decorator from the unittest.mock package you are patching it in the namespace that is under test (in this case app.mocking.get_user_name), not the namespace the function is imported from (in this case app.my_module.get_user_name). To do what you describe with @patch try something like the below: from mock import patch from app.mocking import test_method class MockingTestTestCase(unittest.TestCase): @patch('app.mocking.get_user_name') def test_mock_stubs(self, test_patch): test_patch.return_value = 'Mocked This Silly' ret = test_method() self.assertEqual(ret, 'Mocked This Silly') The standard library documentation includes a useful section describing this.


Instruction:
TensorFlow 2.0 - AttributeError: module 'tensorflow' has no attribute 'Session'

When I am executing the command sess = tf.Session() in a TensorFlow 2.0 environment, I am getting an error message as below: Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: module 'tensorflow' has no attribute 'Session' System information: OS platform and distribution: Windows 10 Python version: 3.7.1 TensorFlow version: 2.0.0-alpha0 (installed with pip) Steps to reproduce: Installation pip install --upgrade pip pip install tensorflow==2.0.0-alpha0 pip install keras pip install numpy==1.16.2 Execution Execute command: import tensorflow as tf Execute command: sess = tf.Session()

Output: According to TF 1:1 Symbols Map, in TensorFlow 2.0, you should use tf.compat.v1.Session() instead of tf.Session(). https://docs.google.com/spreadsheets/d/1FLFJLzg7WNP6JHODX5q8BDgptKafq_slHpnHVbJIteQ/edit#gid=0 To get TensorFlow 1.x-like behaviour in TensorFlow 2.0, one can run import tensorflow.compat.v1 as tf tf.disable_v2_behavior() but then one cannot benefit of many improvements made in TensorFlow 2.0. For more details, please refer to the migration guide.


Instruction:
Cannot install Lxml on Mac OS X 10.9

I want to install Lxml so I can then install Scrapy. When I updated my Mac today it wouldn't let me reinstall lxml, I get the following error: In file included from src/lxml/lxml.etree.c:314: /private/tmp/pip_build_root/lxml/src/lxml/includes/etree_defs.h:9:10: fatal error: 'libxml/xmlversion.h' file not found #include "libxml/xmlversion.h" ^ 1 error generated. error: command 'cc' failed with exit status 1 I have tried using brew to install libxml2 and libxslt, both installed fine but I still cannot install lxml. Last time I was installing I needed to enable the developer tools on Xcode but since it's updated to Xcode 5 it doesn't give me that option anymore. Does anyone know what I need to do?

Output: You should install or upgrade the command line tool for Xcode. Try this in a terminal: xcode-select --install If Xcode Command Line Tools are already installed, but you're still not able to install lxml, then reset xcode-select by following the command: sudo xcode-select --reset This will reset the path to the Xcode Command Line Tools directory and may resolve the issue. After you've installed or updated Xcode Command Line Tools, Hopefully, this will resolve the issue and allow you to install the "lxml" package.


Instruction:
How do I get the different parts of a Flask request's url?

I want to detect if the request came from the localhost:5000 or foo.herokuapp.com host and what path was requested. How do I get this information about a Flask request?

Output: You can examine the url through several Request fields: Imagine your application is listening on the following application root: http://www.example.com/myapplication And a user requests the following URI: http://www.example.com/myapplication/foo/page.html?x=y In this case the values of the above mentioned attributes would be the following: path /foo/page.html full_path /foo/page.html?x=y script_root /myapplication base_url http://www.example.com/myapplication/foo/page.html url http://www.example.com/myapplication/foo/page.html?x=y url_root http://www.example.com/myapplication/ You can easily extract the host part with the appropriate splits. An example of using this: from flask import request @app.route('/') def index(): return request.base_url


Instruction:
How can I enable CORS on Django REST Framework

How can I enable CORS on my Django REST Framework? the reference doesn't help much, it says that I can do by a middleware, but how can I do that?

Output: The link you referenced in your question recommends using django-cors-headers, whose documentation says to install the library python -m pip install django-cors-headers and then add it to your installed apps: INSTALLED_APPS = ( ... 'corsheaders', ... ) You will also need to add a middleware class to listen in on responses: MIDDLEWARE = [ ..., 'corsheaders.middleware.CorsMiddleware', 'django.middleware.common.CommonMiddleware', ..., ] and specify domains for CORS, e.g.: CORS_ALLOWED_ORIGINS = [ 'http://localhost:3030', ] Please browse the configuration section of its documentation, paying particular attention to the various CORS_ORIGIN_ settings. You'll need to set some of those based on your needs.


Instruction:
How can I remove the top and right axis?

Instead of the default "boxed" axis style I want to have only the left and bottom axis, i.e.: +------+ | | | | | | ---> | | | | +------+ +------- This should be easy, but I can't find the necessary options in the docs.

Output: This is the suggested Matplotlib 3 solution from the official website HERE: import numpy as np import matplotlib.pyplot as plt x = np.linspace(0, 2*np.pi, 100) y = np.sin(x) ax = plt.subplot(111) ax.plot(x, y) # Hide the right and top spines ax.spines[['right', 'top']].set_visible(False) plt.show()


Instruction:
Flask at first run: Do not use the development server in a production environment

I installed the Flask plugin in PyCharm Community Edition and I just have this simple code in my flask app: from flask import Flask app = Flask(__name__) @app.route('/') def index(): return '
Hello!
' if __name__ == "__main__": app.run(debug=True) And I get this message: WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead * Restarting with stat * Debugger is active! * Debugger PIN: 123-456-789 * Running on http://127.0.0.1:5000/ Why am I getting this error when I run Flask? A previous version of the message read "Do not use the development server in a production environment."

Output: As of Flask 2.2, the development server always shows this warning, it is not possible to disable it. The development server is not intended for use in production. It is not designed to be particularly efficient, stable, or secure. Use a production WSGI server instead. See the deployment docs from Flask for more information. That warning is just a warning though, it's not an error preventing your app from running. If your app isn't working, there's something else wrong with your code. That warning applies to the development server, not Flask itself. The Flask framework is appropriate for any type of application and deployment.


Instruction:
Correct way to pause a Python program

I've been using the input function as a way to pause my scripts: print("something") wait = input("Press Enter to continue.") print("something") Is there a formal way to do this?

Output: It seems fine to me (or raw_input() in Python 2.X). Alternatively, you could use time.sleep() if you want to pause for a certain number of seconds. import time print("something") time.sleep(5.5) # Pause 5.5 seconds print("something")


Instruction:
What's the difference between a Python "property" and "attribute"?

I am generally confused about the difference between a "property" and an "attribute", and I can't find a great resource to concisely detail the differences.

Output: Properties are a special kind of attribute. Basically, when Python encounters the following code: spam = SomeObject() print(spam.eggs) it looks up eggs in SomeObject1, and then examines eggs to see if it has a __get__, __set__, or __delete__ method -- if it does, it's a property, and Python will call the __get__ method (since we were doing lookup) and return whatever that method returns. If it is not a property, then eggs is looked up in spam, and whatever is found there will be returned. More information about Python's data model and descriptors. 1 Many thanks to Robert Seimer for the correction on the lookup sequence.


Instruction:
How can I avoid issues caused by Python's early-bound default parameters (e.g. mutable default arguments "remembering" old data)?

Sometimes it seems natural to have a default parameter which is an empty list. However, Python produces unexpected behavior in these situations. For example, consider this function: def my_func(working_list=[]): working_list.append("a") print(working_list) The first time it is called, the default will work, but calls after that will update the existing list (with one "a" each call) and print the updated version. How can I fix the function so that, if it is called repeatedly without an explicit argument, a new empty list is used each time?

Output: def my_func(working_list=None): if working_list is None: working_list = [] # alternative: # working_list = [] if working_list is None else working_list working_list.append("a") print(working_list) The docs say you should use None as the default and explicitly test for it in the body of the function. Aside x is None is the comparison recommended by PEP 8: Comparisons to singletons like None should always be done with is or is not, never the equality operators. Also, beware of writing if x when you really mean if x is not None [...] See also What is the difference between "is None" and "== None"


Instruction:
When should I (not) want to use pandas apply() in my code?

I have seen many answers posted to questions on Stack Overflow involving the use of the Pandas method apply. I have also seen users commenting under them saying that "apply is slow, and should be avoided". I have read many articles on the topic of performance that explain apply is slow. I have also seen a disclaimer in the docs about how apply is simply a convenience function for passing UDFs (can't seem to find that now). So, the general consensus is that apply should be avoided if possible. However, this raises the following questions: If apply is so bad, then why is it in the API? How and when should I make my code apply-free? Are there ever any situations where apply is good (better than other possible solutions)?

Output: apply, the Convenience Function you Never Needed We start by addressing the questions in the OP, one by one. "If apply is so bad, then why is it in the API?" DataFrame.apply and Series.apply are convenience functions defined on DataFrame and Series object respectively. apply accepts any user defined function that applies a transformation/aggregation on a DataFrame. apply is effectively a silver bullet that does whatever any existing pandas function cannot do. Some of the things apply can do: Run any user-defined function on a DataFrame or Series Apply a function either row-wise (axis=1) or column-wise (axis=0) on a DataFrame Perform index alignment while applying the function Perform aggregation with user-defined functions (however, we usually prefer agg or transform in these cases) Perform element-wise transformations Broadcast aggregated results to original rows (see the result_type argument). Accept positional/keyword arguments to pass to the user-defined functions. ...Among others. For more information, see Row or Column-wise Function Application in the documentation. So, with all these features, why is apply bad? It is because apply is slow. Pandas makes no assumptions about the nature of your function, and so iteratively applies your function to each row/column as necessary. Additionally, handling all of the situations above means apply incurs some major overhead at each iteration. Further, apply consumes a lot more memory, which is a challenge for memory bounded applications. There are very few situations where apply is appropriate to use (more on that below). If you're not sure whether you should be using apply, you probably shouldn't. pandas 2.2 update: apply now supports engine='numba' More info in the release notes as well as GH54666 Choose between the python (default) engine or the numba engine in apply. The numba engine will attempt to JIT compile the passed function, which may result in speedups for large DataFrames. It also supports the following engine_kwargs : nopython (compile the function in nopython mode) nogil (release the GIL inside the JIT compiled function) parallel (try to apply the function in parallel over the DataFrame) Note: Due to limitations within numba/how pandas interfaces with numba, you should only use this if raw=True Let's address the next question. "How and when should I make my code apply-free?" To rephrase, here are some common situations where you will want to get rid of any calls to apply. Numeric Data If you're working with numeric data, there is likely already a vectorized cython function that does exactly what you're trying to do (if not, please either ask a question on Stack Overflow or open a feature request on GitHub). Contrast the performance of apply for a simple addition operation. df = pd.DataFrame({"A": [9, 4, 2, 1], "B": [12, 7, 5, 4]}) df A B 0 9 12 1 4 7 2 2 5 3 1 4 <!- -> df.apply(np.sum) A 16 B 28 dtype: int64 df.sum() A 16 B 28 dtype: int64 Performance wise, there's no comparison, the cythonized equivalent is much faster. There's no need for a graph, because the difference is obvious even for toy data. %timeit df.apply(np.sum) %timeit df.sum() 2.22 ms ± 41.2 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) 471 µs ± 8.16 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) Even if you enable passing raw arrays with the raw argument, it's still twice as slow. %timeit df.apply(np.sum, raw=True) 840 µs ± 691 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) Another example: df.apply(lambda x: x.max() - x.min()) A 8 B 8 dtype: int64 df.max() - df.min() A 8 B 8 dtype: int64 %timeit df.apply(lambda x: x.max() - x.min()) %timeit df.max() - df.min() 2.43 ms ± 450 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) 1.23 ms ± 14.7 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) In general, seek out vectorized alternatives if possible. String/Regex Pandas provides "vectorized" string functions in most situations, but there are rare cases where those functions do not... "apply", so to speak. A common problem is to check whether a value in a column is present in another column of the same row. df = pd.DataFrame({ 'Name': ['mickey', 'donald', 'minnie'], 'Title': ['wonderland', "welcome to donald's castle", 'Minnie mouse clubhouse'], 'Value': [20, 10, 86]}) df Name Value Title 0 mickey 20 wonderland 1 donald 10 welcome to donald's castle 2 minnie 86 Minnie mouse clubhouse This should return the row second and third row, since "donald" and "minnie" are present in their respective "Title" columns. Using apply, this would be done using df.apply(lambda x: x['Name'].lower() in x['Title'].lower(), axis=1) 0 False 1 True 2 True dtype: bool df[df.apply(lambda x: x['Name'].lower() in x['Title'].lower(), axis=1)] Name Title Value 1 donald welcome to donald's castle 10 2 minnie Minnie mouse clubhouse 86 However, a better solution exists using list comprehensions. df[[y.lower() in x.lower() for x, y in zip(df['Title'], df['Name'])]] Name Title Value 1 donald welcome to donald's castle 10 2 minnie Minnie mouse clubhouse 86 <!- -> %timeit df[df.apply(lambda x: x['Name'].lower() in x['Title'].lower(), axis=1)] %timeit df[[y.lower() in x.lower() for x, y in zip(df['Title'], df['Name'])]] 2.85 ms ± 38.4 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) 788 µs ± 16.4 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) The thing to note here is that iterative routines happen to be faster than apply, because of the lower overhead. If you need to handle NaNs and invalid dtypes, you can build on this using a custom function you can then call with arguments inside the list comprehension. For more information on when list comprehensions should be considered a good option, see my writeup: Are for-loops in pandas really bad? When should I care?. Note Date and datetime operations also have vectorized versions. So, for example, you should prefer pd.to_datetime(df['date']), over, say, df['date'].apply(pd.to_datetime). Read more at the docs. A Common Pitfall: Exploding Columns of Lists s = pd.Series([[1, 2]] * 3) s 0 [1, 2] 1 [1, 2] 2 [1, 2] dtype: object People are tempted to use apply(pd.Series). This is horrible in terms of performance. s.apply(pd.Series) 0 1 0 1 2 1 1 2 2 1 2 A better option is to listify the column and pass it to pd.DataFrame. pd.DataFrame(s.tolist()) 0 1 0 1 2 1 1 2 2 1 2 <!- -> %timeit s.apply(pd.Series) %timeit pd.DataFrame(s.tolist()) 2.65 ms ± 294 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) 816 µs ± 40.5 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) Lastly, "Are there any situations where apply is good?" Apply is a convenience function, so there are situations where the overhead is negligible enough to forgive. It really depends on how many times the function is called. Functions that are Vectorized for Series, but not DataFrames What if you want to apply a string operation on multiple columns? What if you want to convert multiple columns to datetime? These functions are vectorized for Series only, so they must be applied over each column that you want to convert/operate on. df = pd.DataFrame( pd.date_range('2018-12-31','2019-01-31', freq='2D').date.astype(str).reshape(-1, 2), columns=['date1', 'date2']) df date1 date2 0 2018-12-31 2019-01-02 1 2019-01-04 2019-01-06 2 2019-01-08 2019-01-10 3 2019-01-12 2019-01-14 4 2019-01-16 2019-01-18 5 2019-01-20 2019-01-22 6 2019-01-24 2019-01-26 7 2019-01-28 2019-01-30 df.dtypes date1 object date2 object dtype: object This is an admissible case for apply: df.apply(pd.to_datetime, errors='coerce').dtypes date1 datetime64[ns] date2 datetime64[ns] dtype: object Note that it would also make sense to stack, or just use an explicit loop. All these options are slightly faster than using apply, but the difference is small enough to forgive. %timeit df.apply(pd.to_datetime, errors='coerce') %timeit pd.to_datetime(df.stack(), errors='coerce').unstack() %timeit pd.concat([pd.to_datetime(df[c], errors='coerce') for c in df], axis=1) %timeit for c in df.columns: df[c] = pd.to_datetime(df[c], errors='coerce') 5.49 ms ± 247 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) 3.94 ms ± 48.1 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) 3.16 ms ± 216 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) 2.41 ms ± 1.71 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) You can make a similar case for other operations such as string operations, or conversion to category. u = df.apply(lambda x: x.str.contains(...)) v = df.apply(lambda x: x.astype(category)) v/s u = pd.concat([df[c].str.contains(...) for c in df], axis=1) v = df.copy() for c in df: v[c] = df[c].astype(category) And so on... Converting Series to str: astype versus apply This seems like an idiosyncrasy of the API. Using apply to convert integers in a Series to string is comparable (and sometimes faster) than using astype. The graph was plotted using the perfplot library. import perfplot perfplot.show( setup=lambda n: pd.Series(np.random.randint(0, n, n)), kernels=[ lambda s: s.astype(str), lambda s: s.apply(str) ], labels=['astype', 'apply'], n_range=[2**k for k in range(1, 20)], xlabel='N', logx=True, logy=True, equality_check=lambda x, y: (x == y).all()) With floats, I see the astype is consistently as fast as, or slightly faster than apply. So this has to do with the fact that the data in the test is integer type. GroupBy operations with chained transformations GroupBy.apply has not been discussed until now, but GroupBy.apply is also an iterative convenience function to handle anything that the existing GroupBy functions do not. One common requirement is to perform a GroupBy and then two prime operations such as a "lagged cumsum": df = pd.DataFrame({"A": list('aabcccddee'), "B": [12, 7, 5, 4, 5, 4, 3, 2, 1, 10]}) df A B 0 a 12 1 a 7 2 b 5 3 c 4 4 c 5 5 c 4 6 d 3 7 d 2 8 e 1 9 e 10 <!- -> You'd need two successive groupby calls here: df.groupby('A').B.cumsum().groupby(df.A).shift() 0 NaN 1 12.0 2 NaN 3 NaN 4 4.0 5 9.0 6 NaN 7 3.0 8 NaN 9 1.0 Name: B, dtype: float64 Using apply, you can shorten this to a a single call. df.groupby('A').B.apply(lambda x: x.cumsum().shift()) 0 NaN 1 12.0 2 NaN 3 NaN 4 4.0 5 9.0 6 NaN 7 3.0 8 NaN 9 1.0 Name: B, dtype: float64 It is very hard to quantify the performance because it depends on the data. But in general, apply is an acceptable solution if the goal is to reduce a groupby call (because groupby is also quite expensive). Other Caveats Aside from the caveats mentioned above, it is also worth mentioning that apply operates on the first row (or column) twice. This is done to determine whether the function has any side effects. If not, apply may be able to use a fast-path for evaluating the result, else it falls back to a slow implementation. df = pd.DataFrame({ 'A': [1, 2], 'B': ['x', 'y'] }) def func(x): print(x['A']) return x df.apply(func, axis=1) # 1 # 1 # 2 A B 0 1 x 1 2 y This behaviour is also seen in GroupBy.apply on pandas versions <0.25 (it was fixed for 0.25, see here for more information.)


Instruction:
NumPy or Pandas: Keeping array type as integer while having a NaN value

Is there a preferred way to keep the data type of a numpy array fixed as int (or int64 or whatever), while still having an element inside listed as numpy.NaN? In particular, I am converting an in-house data structure to a Pandas DataFrame. In our structure, we have integer-type columns that still have NaN's (but the dtype of the column is int). It seems to recast everything as a float if we make this a DataFrame, but we'd really like to be int. Thoughts? Things tried: I tried using the from_records() function under pandas.DataFrame, with coerce_float=False and this did not help. I also tried using NumPy masked arrays, with NaN fill_value, which also did not work. All of these caused the column data type to become a float.

Output: This capability has been added to pandas beginning with version 0.24. At this point, it requires the use of extension dtype 'Int64' (capitalized), rather than the default dtype 'int64' (lowercase).


Instruction:
python date of the previous month

I am trying to get the date of the previous month with python. Here is what i've tried: str( time.strftime('%Y') ) + str( int(time.strftime('%m'))-1 ) However, this way is bad for 2 reasons: First it returns 20122 for the February of 2012 (instead of 201202) and secondly it will return 0 instead of 12 on January. I have solved this trouble in bash with echo $(date -d"3 month ago" "+%G%m%d") I think that if bash has a built-in way for this purpose, then python, much more equipped, should provide something better than forcing writing one's own script to achieve this goal. Of course i could do something like: if int(time.strftime('%m')) == 1: return '12' else: if int(time.strftime('%m')) < 10: return '0'+str(time.strftime('%m')-1) else: return str(time.strftime('%m') -1) I have not tested this code and i don't want to use it anyway (unless I can't find any other way:/) Thanks for your help!

Output: datetime and the datetime.timedelta classes are your friend. find today. use that to find the first day of this month. use timedelta to backup a single day, to the last day of the previous month. print the YYYYMM string you're looking for. Like this: import datetime today = datetime.date.today() first = today.replace(day=1) last_month = first - datetime.timedelta(days=1) print(last_month.strftime("%Y%m")) 201202 is printed.


Instruction:
How to get current available GPUs in tensorflow?

I have a plan to use distributed TensorFlow, and I saw TensorFlow can use GPUs for training and testing. In a cluster environment, each machine could have 0 or 1 or more GPUs, and I want to run my TensorFlow graph into GPUs on as many machines as possible. I found that when running tf.Session() TensorFlow gives information about GPU in the log messages like below: I tensorflow/core/common_runtime/gpu/gpu_init.cc:126] DMA: 0 I tensorflow/core/common_runtime/gpu/gpu_init.cc:136] 0: Y I tensorflow/core/common_runtime/gpu/gpu_device.cc:838] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1080, pci bus id: 0000:01:00.0) My question is how do I get information about current available GPU from TensorFlow? I can get loaded GPU information from the log, but I want to do it in a more sophisticated, programmatic way. I also could restrict GPUs intentionally using the CUDA_VISIBLE_DEVICES environment variable, so I don't want to know a way of getting GPU information from OS kernel. In short, I want a function like tf.get_available_gpus() that will return ['/gpu:0', '/gpu:1'] if there are two GPUs available in the machine. How can I implement this?

Output: Since TensorFlow 2.1, you can use tf.config.list_physical_devices('GPU'): import tensorflow as tf gpus = tf.config.list_physical_devices('GPU') for gpu in gpus: print("Name:", gpu.name, " Type:", gpu.device_type) If you have two GPUs installed, it outputs this: Name: /physical_device:GPU:0 Type: GPU Name: /physical_device:GPU:1 Type: GPU In TF 2.0, you must add experimental: gpus = tf.config.experimental.list_physical_devices('GPU') See: Guide pages Current API


Instruction:
Install a module using pip for specific python version

On Ubuntu 10.04 by default Python 2.6 is installed, then I have installed Python 2.7. How can I use pip install to install packages for Python 2.7. For example: pip install beautifulsoup4 by default installs BeautifulSoup for Python 2.6 When I do: import bs4 in Python 2.6 it works, but in Python 2.7 it says: No module named bs4

Output: Use a version of pip installed against the Python instance you want to install new packages to. In many distributions, there may be separate python2.6-pip and python2.7-pip packages, invoked with binary names such as pip-2.6 and pip-2.7. If pip is not packaged in your distribution for the desired target, you might look for a setuptools or easyinstall package, or use virtualenv (which will always include pip in a generated environment). pip's website includes installation instructions, if you can't find anything within your distribution. Update: In the modern (2020s) world, python -m pip is the right way to do this, where python can be replaced with whichever interpreter you choose.


Instruction:
Parse date string and change format

I have a date string with the format 'Mon Feb 15 2010'. I want to change the format to '15/02/2010'. How can I do this?

Output: The datetime module could help you with that: datetime.datetime.strptime(input_date_string, input_format).strftime(output_format) For the specific example, you could do: >>> from datetime import datetime >>> datetime.strptime('Mon Feb 15 2010', '%a %b %d %Y').strftime('%d/%m/%Y') '15/02/2010' Learn more about different formats here.


Instruction:
Putting an if-elif-else statement on one line?

Is there an easier way of writing an if-elif-else statement so it fits on one line? For example, if expression1: statement1 elif expression2: statement2 else: statement3 Or a real-world example: if i > 100: x = 2 elif i < 100: x = 1 else: x = 0 I just feel if the example above could be written the following way, it could look like more concise. x = 2 if i>100 elif i<100 1 else 0 # [WRONG] I have read the link below, but it doesn't address my question. Does Python have a ternary conditional operator? (the question is about condensing an if-else statement to one line)

Output: No, it's not possible (at least not with arbitrary statements), nor is it desirable. Fitting everything on one line would most likely violate PEP-8 where it is mandated that lines should not exceed 80 characters in length. It's also against the Zen of Python: "Readability counts". (Type import this at the Python prompt to read the whole thing). You can use a ternary expression in Python, but only for expressions, not for statements: >>> a = "Hello" if foo() else "Goodbye" Edit: Your revised question now shows that the three statements are identical except for the value being assigned. In that case, a chained ternary operator does work, but I still think that it's less readable: >>> i = 100 >>> x = 2 if i>100 else 1 if i<100 else 0 >>> x 0 >>> i = 101 >>> x = 2 if i>100 else 1 if i<100 else 0 >>> x 2 >>> i = 99 >>> x = 2 if i>100 else 1 if i<100 else 0 >>> x 1


Instruction:
Deep merge dictionaries of dictionaries in Python

I need to merge multiple dictionaries, here's what I have for instance: dict1 = {1:{"a":{"A"}}, 2:{"b":{"B"}}} dict2 = {2:{"c":{"C"}}, 3:{"d":{"D"}}} With A B C and D being leaves of the tree, like {"info1":"value", "info2":"value2"} There is an unknown level(depth) of dictionaries, it could be {2:{"c":{"z":{"y":{C}}}}} In my case it represents a directory/files structure with nodes being docs and leaves being files. I want to merge them to obtain: dict3 = {1:{"a":{"A"}}, 2:{"b":{"B"},"c":{"C"}}, 3:{"d":{"D"}}} I'm not sure how I could do that easily with Python.

Output: This is actually quite tricky - particularly if you want a useful error message when things are inconsistent, while correctly accepting duplicate but consistent entries (something no other answer here does..) Assuming you don't have huge numbers of entries, a recursive function is easiest: def merge(a: dict, b: dict, path=[]): for key in b: if key in a: if isinstance(a[key], dict) and isinstance(b[key], dict): merge(a[key], b[key], path + [str(key)]) elif a[key] != b[key]: raise Exception('Conflict at ' + '.'.join(path + [str(key)])) else: a[key] = b[key] return a # works print(merge({1:{"a":"A"},2:{"b":"B"}}, {2:{"c":"C"},3:{"d":"D"}})) # has conflict merge({1:{"a":"A"},2:{"b":"B"}}, {1:{"a":"A"},2:{"b":"C"}}) note that this mutates a - the contents of b are added to a (which is also returned). If you want to keep a you could call it like merge(dict(a), b). agf pointed out (below) that you may have more than two dicts, in which case you can use: from functools import reduce reduce(merge, [dict1, dict2, dict3...]) where everything will be added to dict1. Note: I edited my initial answer to mutate the first argument; that makes the "reduce" easier to explain


Instruction:
Wrap long lines in Python

How do I wrap long lines in Python without sacrificing indentation? For example: def fun(): print '{0} Here is a really long sentence with {1}'.format(3, 5) Suppose this goes over the 79 character recommended limit. The way I read it, here is how to indent it: def fun(): print '{0} Here is a really long \ sentence with {1}'.format(3, 5) However, with this approach, the indentation of the continued line matches the indentation of the fun(). This looks kinda ugly. If someone was to go through my code, it would look bad to have uneven indentation because of this print statement. How do I indent lines like this effectively without sacrificing code readability?

Output: def fun(): print(('{0} Here is a really long ' 'sentence with {1}').format(3, 5)) Adjacent string literals are concatenated at compile time, just as in C. 2.4.2. String literal concatenation is a good place to start for more information.


Instruction:
How do I turn a python datetime into a string, with readable format date?

t = e['updated_parsed'] dt = datetime.datetime(t[0],t[1],t[2],t[3],t[4],t[5],t[6] print dt >>>2010-01-28 08:39:49.000003 How do I turn that into a string?: "January 28, 2010"

Output: The datetime class has a method strftime. The Python docs documents the different formats it accepts: strftime() and strptime() Behavior For this specific example, it would look something like: my_datetime.strftime("%B %d, %Y")


Instruction:
What does model.train() do in PyTorch?

Does it call forward() in nn.Module? I thought when we call the model, forward method is being used. Why do we need to specify train()?

Output: model.train() tells your model that you are training the model. This helps inform layers such as Dropout and BatchNorm, which are designed to behave differently during training and evaluation. For instance, in training mode, BatchNorm updates a moving average on each new batch; whereas, for evaluation mode, these updates are frozen. More details: model.train() sets the mode to train (see source code). You can call either model.eval() or model.train(mode=False) to tell that you are testing. It is somewhat intuitive to expect train function to train model but it does not do that. It just sets the mode.