Instruction:
How to define an empty generator function?

A generator function can be defined by putting the yield keyword in the function’s body: def gen(): for i in range(10): yield i How to define an empty generator function? The following code doesn’t work, since Python cannot know that it is supposed to be a generator function instead of a normal function: def empty(): pass I could do something like this: def empty(): if False: yield But that would be very ugly. Is there a nicer way?

Output: You can use return once in a generator; it stops iteration without yielding anything, and thus provides an explicit alternative to letting the function run out of scope. So use yield to turn the function into a generator, but precede it with return to terminate the generator before yielding anything. >>> def f(): ... return ... yield ... >>> list(f()) [] I'm not sure it's that much better than what you have -- it just replaces a no-op if statement with a no-op yield statement. But it is more idiomatic. Note that just using yield doesn't work. >>> def f(): ... yield ... >>> list(f()) [None] Why not just use iter(())? This question asks specifically about an empty generator function. For that reason, I take it to be a question about the internal consistency of Python's syntax, rather than a question about the best way to create an empty iterator in general. If question is actually about the best way to create an empty iterator, then you might agree with Zectbumo about using iter(()) instead. However, it's important to observe that iter(()) doesn't return a function! It directly returns an empty iterable. Suppose you're working with an API that expects a callable that returns an iterable each time it's called, just like an ordinary generator function. You'll have to do something like this: def empty(): return iter(()) (Credit should go to Unutbu for giving the first correct version of this answer.) Now, you may find the above clearer, but I can imagine situations in which it would be less clear. Consider this example of a long list of (contrived) generator function definitions: def zeros(): while True: yield 0 def ones(): while True: yield 1 ... At the end of that long list, I'd rather see something with a yield in it, like this: def empty(): return yield or, in Python 3.3 and above (as suggested by DSM), this: def empty(): yield from () The presence of the yield keyword makes it clear at the briefest glance that this is just another generator function, exactly like all the others. It takes a bit more time to see that the iter(()) version is doing the same thing. It's a subtle difference, but I honestly think the yield-based functions are more readable and maintainable. See also this great answer from user3840170 that uses dis to show another reason why this approach is preferable: it emits the fewest instructions when compiled.


Instruction:
Is there a better way to iterate over two lists, getting one element from each list for each iteration?

I have a list of Latitudes and one of Longitudes and need to iterate over the latitude and longitude pairs. Is it better to: A. Assume that the lists are of equal lengths: for i in range(len(Latitudes)): Lat,Long=(Latitudes[i],Longitudes[i]) B. Or: for Lat,Long in [(x,y) for x in Latitudes for y in Longitudes]: (Note that B is incorrect. This gives me all the pairs, equivalent to itertools.product()) Any thoughts on the relative merits of each, or which is more pythonic?

Output: This is as pythonic as you can get: for lat, long in zip(Latitudes, Longitudes): print(lat, long)


Instruction:
Why does the floating-point value of 4*0.1 look nice in Python 3 but 3*0.1 doesn't?

I know that most decimals don't have an exact floating point representation (Is floating point math broken?). But I don't see why 4*0.1 is printed nicely as 0.4, but 3*0.1 isn't, when both values actually have ugly decimal representations: >>> 3*0.1 0.30000000000000004 >>> 4*0.1 0.4 >>> from decimal import Decimal >>> Decimal(3*0.1) Decimal('0.3000000000000000444089209850062616169452667236328125') >>> Decimal(4*0.1) Decimal('0.40000000000000002220446049250313080847263336181640625')

Output: The simple answer is because 3*0.1 != 0.3 due to quantization (roundoff) error (whereas 4*0.1 == 0.4 because multiplying by a power of two is usually an "exact" operation). Python tries to find the shortest string that would round to the desired value, so it can display 4*0.1 as 0.4 as these are equal, but it cannot display 3*0.1 as 0.3 because these are not equal. You can use the .hex method in Python to view the internal representation of a number (basically, the exact binary floating point value, rather than the base-10 approximation). This can help to explain what's going on under the hood. >>> (0.1).hex() '0x1.999999999999ap-4' >>> (0.3).hex() '0x1.3333333333333p-2' >>> (0.1*3).hex() '0x1.3333333333334p-2' >>> (0.4).hex() '0x1.999999999999ap-2' >>> (0.1*4).hex() '0x1.999999999999ap-2' 0.1 is 0x1.999999999999a times 2^-4. The "a" at the end means the digit 10 - in other words, 0.1 in binary floating point is very slightly larger than the "exact" value of 0.1 (because the final 0x0.99 is rounded up to 0x0.a). When you multiply this by 4, a power of two, the exponent shifts up (from 2^-4 to 2^-2) but the number is otherwise unchanged, so 4*0.1 == 0.4. However, when you multiply by 3, the tiny little difference between 0x0.99 and 0x0.a0 (0x0.07) magnifies into a 0x0.15 error, which shows up as a one-digit error in the last position. This causes 0.1*3 to be very slightly larger than the rounded value of 0.3. Python 3's float repr is designed to be round-trippable, that is, the value shown should be exactly convertible into the original value (float(repr(f)) == f for all floats f). Therefore, it cannot display 0.3 and 0.1*3 exactly the same way, or the two different numbers would end up the same after round-tripping. Consequently, Python 3's repr engine chooses to display one with a slight apparent error.


Instruction:
sorting by a custom list in pandas

After reading through: http://pandas.pydata.org/pandas-docs/version/0.13.1/generated/pandas.DataFrame.sort.html I still can't seem to figure out how to sort a column by a custom list. Obviously, the default sort is alphabetical. I'll give an example. Here is my (very abridged) dataframe: Player Year Age Tm G 2967 Cedric Hunter 1991 27 CHH 6 5335 Maurice Baker 2004 25 VAN 7 13950 Ratko Varda 2001 22 TOT 60 6141 Ryan Bowen 2009 34 OKC 52 6169 Adrian Caldwell 1997 31 DAL 81 I want to be able to sort by Player, Year and then Tm. The default sort by Player and Year is fine for me, in normal order. However, I do not want Team sorted alphabetically b/c I want TOT always at the top. Here is the list I created: sorter = ['TOT', 'ATL', 'BOS', 'BRK', 'CHA', 'CHH', 'CHI', 'CLE', 'DAL', 'DEN', 'DET', 'GSW', 'HOU', 'IND', 'LAC', 'LAL', 'MEM', 'MIA', 'MIL', 'MIN', 'NJN', 'NOH', 'NOK', 'NOP', 'NYK', 'OKC', 'ORL', 'PHI', 'PHO', 'POR', 'SAC', 'SAS', 'SEA', 'TOR', 'UTA', 'VAN', 'WAS', 'WSB'] After reading through the link above, I thought this would work but it didn't: df.sort(['Player', 'Year', 'Tm'], ascending = [True, True, sorter]) It still has ATL at the top, meaning that it sorted alphabetically and not according to my custom list. Any help would really be greatly appreciated, I just can't figure this out.

Output: Below is an example that performs lexicographic sort on a dataframe. The idea is to create an numerical index based on the specific sort. Then to perform a numerical sort based on the index. A column is added to the dataframe to do so, and is then removed. import pandas as pd # Create DataFrame df = pd.DataFrame( {'id':[2967, 5335, 13950, 6141, 6169], 'Player': ['Cedric Hunter', 'Maurice Baker', 'Ratko Varda' ,'Ryan Bowen' ,'Adrian Caldwell'], 'Year': [1991, 2004, 2001, 2009, 1997], 'Age': [27, 25, 22, 34, 31], 'Tm': ['CHH' ,'VAN' ,'TOT' ,'OKC', 'DAL'], 'G': [6, 7, 60, 52, 81]}) # Define the sorter sorter = ['TOT', 'ATL', 'BOS', 'BRK', 'CHA', 'CHH', 'CHI', 'CLE', 'DAL','DEN', 'DET', 'GSW', 'HOU', 'IND', 'LAC', 'LAL', 'MEM', 'MIA', 'MIL', 'MIN', 'NJN', 'NOH', 'NOK', 'NOP', 'NYK', 'OKC', 'ORL', 'PHI', 'PHO', 'POR', 'SAC', 'SAS', 'SEA', 'TOR', 'UTA', 'VAN', 'WAS', 'WSB'] # Create the dictionary that defines the order for sorting sorterIndex = dict(zip(sorter, range(len(sorter)))) # Generate a rank column that will be used to sort # the dataframe numerically df['Tm_Rank'] = df['Tm'].map(sorterIndex) # Here is the result asked with the lexicographic sort # Result may be hard to analyze, so a second sorting is # proposed next ## NOTE: ## Newer versions of pandas use 'sort_values' instead of 'sort' df.sort_values(['Player', 'Year', 'Tm_Rank'], ascending = [True, True, True], inplace = True) df.drop('Tm_Rank', 1, inplace = True) print(df) # Here is an example where 'Tm' is sorted first, that will # give the first row of the DataFrame df to contain TOT as 'Tm' df['Tm_Rank'] = df['Tm'].map(sorterIndex) ## NOTE: ## Newer versions of pandas use 'sort_values' instead of 'sort' df.sort_values(['Tm_Rank', 'Player', 'Year'], ascending = [True , True, True], inplace = True) df.drop('Tm_Rank', 1, inplace = True) print(df)


Instruction:
Filtering a list of strings based on contents

Given the list ['a','ab','abc','bac'], I want to compute a list with strings that have 'ab' in them. I.e. the result is ['ab','abc']. How can this be done in Python?

Output: This simple filtering can be achieved in many ways with Python. The best approach is to use "list comprehensions" as follows: >>> lst = ['a', 'ab', 'abc', 'bac'] >>> [k for k in lst if 'ab' in k] ['ab', 'abc'] Another way is to use the filter function. In Python 2: >>> filter(lambda k: 'ab' in k, lst) ['ab', 'abc'] In Python 3, it returns an iterator instead of a list, but you can cast it: >>> list(filter(lambda k: 'ab' in k, lst)) ['ab', 'abc'] Though it's better practice to use a comprehension.


Instruction:
Could not load dynamic library 'cudart64_101.dll' on tensorflow CPU-only installation

I just installed the latest version of Tensorflow via pip install tensorflow and whenever I run a program, I get the log message: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_101.dll'; dlerror: cudart64_101.dll not found Is this bad? How do I fix the error?

Output: Tensorflow 2.1+ What's going on? With the new Tensorflow 2.1 release, the default tensorflow pip package contains both CPU and GPU versions of TF. In previous TF versions, not finding the CUDA libraries would emit an error and raise an exception, while now the library dynamically searches for the correct CUDA version and, if it doesn't find it, emits the warning (The W in the beginning stands for warnings, errors have an E (or F for fatal errors) and falls back to CPU-only mode. In fact, this is also written in the log as an info message right after the warning (do note that if you have a higher minimum log level that the default, you might not see info messages). The full log is (emphasis mine): 2020-01-20 12:27:44.554767: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_101.dll'; dlerror: cudart64_101.dll not found 2020-01-20 12:27:44.554964: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine. Should I worry? How do I fix it? If you don't have a CUDA-enabled GPU on your machine, or if you don't care about not having GPU acceleration, no need to worry. If, on the other hand, you installed tensorflow and wanted GPU acceleration, check your CUDA installation (TF 2.1 requires CUDA 10.1, not 10.2 or 10.0). If you just want to get rid of the warning, you can adapt TF's logging level to suppress warnings, but that might be overkill, as it will silence all warnings. Tensorflow 1.X or 2.0: Your CUDA setup is broken, ensure you have the correct version installed.


Instruction:
How do I activate a virtualenv inside PyCharm's terminal?

I've set up PyCharm, created my virtualenv (either through the virtual env command, or directly in PyCharm) and activated that environment as my Interpreter. Everything is working just fine. However, if I open a terminal using "Tools, Open Terminal", the shell prompt supplied is not using the virtual env; I still have to use source ~/envs/someenv/bin/activate within that Terminal to activate it. Another method is to activate the environment in a shell, and run PyCharm from that environment. This is "workable" but pretty ugly, and means I have major problems if I switch environments or projects from PyCharm: I'm now using the totally-wrong environment. Is there some other, much-easier way to have "Tools, Open Terminal" automatically activate the virtual environment?

Output: Edit: According to https://www.jetbrains.com/pycharm/whatsnew/#v2016-3-venv-in-terminal, PyCharm 2016.3 (released Nov 2016) has virutalenv support for terminals out of the box Auto virtualenv is supported for bash, zsh, fish, and Windows cmd. You can customize your shell preference in Settings (Preferences) | Tools | Terminal | check Activate virtaulenv you also need to make sure to have the path of virtual environment path included in the content root folder of your project structure. You can go to settings (preference) | project | Project Structure | if your environment is not included in the project directory. ***Old Method:*** Create a file .pycharmrc in your home folder with the following contents source ~/.bashrc source ~/pycharmvenv/bin/activate Use your virtualenv path as the last parameter. Then set the shell Preferences->Project Settings->Shell path to /bin/bash --rcfile ~/.pycharmrc


Instruction:
Fill between two vertical lines

I went through the examples in the matplotlib documentation, but it wasn't clear to me how I can make a plot that fills the area between two specific vertical lines. For example, say I want to create a plot between x=0.2 and x=4 (for the full y range of the plot). Should I use fill_between, fill or fill_betweenx? Can I use the where condition for this?

Output: It sounds like you want axvspan, rather than one of the fill between functions. The differences is that axvspan (and axhspan) will fill up the entire y (or x) extent of the plot regardless of how you zoom. For example, let's use axvspan to highlight the x-region between 8 and 14: import matplotlib.pyplot as plt fig, ax = plt.subplots() ax.plot(range(20)) ax.axvspan(8, 14, alpha=0.5, color='red') plt.show() You could use fill_betweenx to do this, but the extents (both x and y) of the rectangle would be in data coordinates. With axvspan, the y-extents of the rectangle default to 0 and 1 and are in axes coordinates (in other words, percentages of the height of the plot). To illustrate this, let's make the rectangle extend from 10% to 90% of the height (instead of taking up the full extent). Try zooming or panning, and notice that the y-extents say fixed in display space, while the x-extents move with the zoom/pan: import matplotlib.pyplot as plt fig, ax = plt.subplots() ax.plot(range(20)) ax.axvspan(8, 14, ymin=0.1, ymax=0.9, alpha=0.5, color='red') plt.show()


Instruction:
How to use %f with strftime() in Python to get microseconds?

I'm trying to use strftime() to microsecond precision, which seems possible using %f (as stated here). However when I try the following code: import time import strftime from time print strftime("%H:%M:%S.%f") ...I get the hour, the minutes and the seconds, but %f prints as %f, with no sign of the microseconds. I'm running Python 2.6.5 on Ubuntu, so it should be fine and %f should be supported (it's supported for 2.6 and above, as far as I know.)

Output: You can use datetime's strftime() function to get this. The problem is that time's strftime() accepts a timetuple that does not carry microsecond information. from datetime import datetime datetime.now().strftime("%H:%M:%S.%f") Should do the trick!


Instruction:
asyncio.sleep() vs time.sleep()

When I go to the asyncio page, the first example is a hello world program. When I run it on python 3.73, I can't see any different from the normal one. can anyone tell me the difference and give a non-trivial example? In [1]: import asyncio ...: ...: async def main(): ...: print('Hello ...') ...: await asyncio.sleep(5) ...: print('... World!') ...: ...: # Python 3.7+ ...: asyncio.run(main()) Hello ... ... World! In [2]: In [2]: import time ...: ...: def main(): ...: print('Hello ...') ...: time.sleep(5) ...: print('... World!') ...: ...: # Python 3.7+ ...: main() Hello ... ... World! I intentionally increase the time from 1s to 5s, hope to see something special but I didn't.

Output: You aren't seeing anything special because there's nothing much asynchronous work in your code. However, the main difference is that time.sleep(5) is blocking, and asyncio.sleep(5) is non-blocking. When time.sleep(5) is called, it will block the entire execution of the script and it will be put on hold, just frozen, doing nothing. But when you call await asyncio.sleep(5), it will ask the event loop to run something else while your await statement finishes its execution. Here's an improved example. import asyncio async def hello(): print('Hello ...') await asyncio.sleep(1) print('... World!') async def main(): await asyncio.gather(hello(), hello()) asyncio.run(main()) Will output: ~$ python3.7 async.py Hello ... Hello ... ... World! ... World! You can see that await asyncio.sleep(1) is not blocking the execution of the script. In contrast, replacing the line await asyncio.sleep(1) with time.sleep(1), the output will be Hello ... ... World! Hello ... ... World! because time.sleep is blocking and the first call of hello() has to finish first before the second call of hello() starts running.


Instruction:
How to use multiprocessing queue in Python?

I'm having much trouble trying to understand just how the multiprocessing queue works on python and how to implement it. Lets say I have two python modules that access data from a shared file, let's call these two modules a writer and a reader. My plan is to have both the reader and writer put requests into two separate multiprocessing queues, and then have a third process pop these requests in a loop and execute as such. My main problem is that I really don't know how to implement multiprocessing.queue correctly, you cannot really instantiate the object for each process since they will be separate queues, how do you make sure that all processes relate to a shared queue (or in this case, queues)

Output: Short Summary As of CY2023, the technique described in this answer is quite out of date. These days, use concurrent.futures.ProcessPoolExecutor() instead of multiprocessing, below... This answer describes the benefits and shortcomings of using concurrent.futures.ProcessPoolExecutor(). FYI, multiple python processes are sometimes used instead of threading to get the most benefit from concurrency. That said, python threading works pretty well as long as there is sufficient CPU activity to avoid the GIL (activity such as sending / receiving network traffic). Original Answer My main problem is that I really don't know how to implement multiprocessing.queue correctly, you cannot really instantiate the object for each process since they will be separate queues, how do you make sure that all processes relate to a shared queue (or in this case, queues) This is a simple example of a reader and writer sharing a single queue... The writer sends a bunch of integers to the reader; when the writer runs out of numbers, it sends 'DONE', which lets the reader know to break out of the read loop. You can spawn as many reader processes as you like... from multiprocessing import Process, Queue import time import sys def reader_proc(queue): """Read from the queue; this spawns as a separate Process""" while True: msg = queue.get() # Read from the queue and do nothing if msg == "DONE": break def writer(count, num_of_reader_procs, queue): """Write integers into the queue. A reader_proc() will read them from the queue""" for ii in range(0, count): queue.put(ii) # Put 'count' numbers into queue ### Tell all readers to stop... for ii in range(0, num_of_reader_procs): queue.put("DONE") def start_reader_procs(qq, num_of_reader_procs): """Start the reader processes and return all in a list to the caller""" all_reader_procs = list() for ii in range(0, num_of_reader_procs): ### reader_p() reads from qq as a separate process... ### you can spawn as many reader_p() as you like ### however, there is usually a point of diminishing returns reader_p = Process(target=reader_proc, args=((qq),)) reader_p.daemon = True reader_p.start() # Launch reader_p() as another proc all_reader_procs.append(reader_p) return all_reader_procs if __name__ == "__main__": num_of_reader_procs = 2 qq = Queue() # writer() writes to qq from _this_ process for count in [10**4, 10**5, 10**6]: assert 0 < num_of_reader_procs < 4 all_reader_procs = start_reader_procs(qq, num_of_reader_procs) writer(count, len(all_reader_procs), qq) # Queue stuff to all reader_p() print("All reader processes are pulling numbers from the queue...") _start = time.time() for idx, a_reader_proc in enumerate(all_reader_procs): print(" Waiting for reader_p.join() index %s" % idx) a_reader_proc.join() # Wait for a_reader_proc() to finish print(" reader_p() idx:%s is done" % idx) print( "Sending {0} integers through Queue() took {1} seconds".format( count, (time.time() - _start) ) ) print("")


Instruction:
Pandas "Can only compare identically-labeled DataFrame objects" error

I'm using Pandas to compare the outputs of two files loaded into two data frames (uat, prod): ... uat = uat[['Customer Number','Product']] prod = prod[['Customer Number','Product']] print uat['Customer Number'] == prod['Customer Number'] print uat['Product'] == prod['Product'] print uat == prod The first two match exactly: 74357 True 74356 True Name: Customer Number, dtype: bool 74357 True 74356 True Name: Product, dtype: bool For the third print, I get an error: Can only compare identically-labeled DataFrame objects. If the first two compared fine, what's wrong with the 3rd? Thanks

Output: Here's a small example to demonstrate this (which only applied to DataFrames, not Series, until Pandas 0.19 where it applies to both): In [1]: df1 = pd.DataFrame([[1, 2], [3, 4]]) In [2]: df2 = pd.DataFrame([[3, 4], [1, 2]], index=[1, 0]) In [3]: df1 == df2 Exception: Can only compare identically-labeled DataFrame objects One solution is to sort the index first (Note: some functions require sorted indexes): In [4]: df2.sort_index(inplace=True) In [5]: df1 == df2 Out[5]: 0 1 0 True True 1 True True Note: == is also sensitive to the order of columns, so you may have to use sort_index(axis=1): In [11]: df1.sort_index().sort_index(axis=1) == df2.sort_index().sort_index(axis=1) Out[11]: 0 1 0 True True 1 True True Note: This can still raise (if the index/columns aren't identically labelled after sorting).


Instruction:
Why does the repr() of a string have two sets of quotes? And why don't I get the object back when I eval() it?

repr(): evaluatable string representation of an object (you can eval() it, meaning it is a string representation that evaluates to a Python object) In other words: >>> x = 'foo' >>> repr(x) "'foo'" Questions: Why do I get the double quotes when I do repr(x)? (I don't get them when I do str(x)) Why do I get 'foo' when I do eval("'foo'") and not x which is the object?

Output: Consider: >>> x = 'foo' >>> x 'foo' So the name x is attached to 'foo' string. When you call for example repr(x), the interpreter puts 'foo' instead of x and then calls repr('foo'). >>> repr(x) "'foo'" >>> x.__repr__() "'foo'" repr actually calls a magic method, __repr__, of x, which gives the string containing the representation of the value 'foo' assigned to x. So it returns 'foo' inside the string "" resulting in "'foo'". The idea of repr is to give a string which contains a series of symbols which we can type in the interpreter and get the same value which was sent as an argument to repr. >>> eval("'foo'") 'foo' When we call eval("'foo'"), it's the same as if we type 'foo' in the interpreter. It's as if we directly type the contents of the outer string "" in the interpreter. >>> eval('foo') Traceback (most recent call last): File "<pyshell#5>", line 1, in <module> eval('foo') File "<string>", line 1, in <module> NameError: name 'foo' is not defined If we call eval('foo'), it's the same as if we type foo in the interpreter. But there is no foo variable available and an exception is raised. >>> str(x) 'foo' >>> x.__str__() 'foo' >>> str is just the string representation of the object (remember, x variable refers to 'foo'), so this function returns string. >>> str(5) '5' The string representation of integer 5 is '5'. >>> str('foo') 'foo' And the string representation of string 'foo' is the same string 'foo'.


Instruction:
Python: Figure out local timezone

I want to compare UTC timestamps from a log file with local timestamps. When creating the local datetime object, I use something like: >>> local_time=datetime.datetime(2010, 4, 27, 12, 0, 0, 0, tzinfo=pytz.timezone('Israel')) I want to find an automatic tool that would replace thetzinfo=pytz.timezone('Israel') with the current local time zone. Any ideas?

Output: Try dateutil, which has a tzlocal type that does what you need.


Instruction:
How to set and get cookies in Django?

I have a web site which shows different content based on a location the visitor chooses. e.g: User enters in 55812 as the zip. I know what city and area lat/long. that is and give them their content pertinent to that area. My question is how can I store this in a cookie so that when they return they are not required to always enter their zip code? I see it as follows: Set persistent cookie based on their area. When they return read cookie, grab zipcode. Return content based on the zip code in their cookie. I can't seem to find any solid information on setting a cookie. Any help is greatly appreciated.

Output: UPDATE : check Peter's answer below for a builtin solution : This is a helper to set a persistent cookie: import datetime def set_cookie(response, key, value, days_expire=7): if days_expire is None: max_age = 365 * 24 * 60 * 60 # one year else: max_age = days_expire * 24 * 60 * 60 expires = datetime.datetime.strftime( datetime.datetime.utcnow() + datetime.timedelta(seconds=max_age), "%a, %d-%b-%Y %H:%M:%S GMT", ) response.set_cookie( key, value, max_age=max_age, expires=expires, domain=settings.SESSION_COOKIE_DOMAIN, secure=settings.SESSION_COOKIE_SECURE or None, ) Use the following code before sending a response. def view(request): response = HttpResponse("hello") set_cookie(response, 'name', 'jujule') return response UPDATE : check Peter's answer below for a builtin solution :


Instruction:
heapq with custom compare predicate

I am trying to build a heap with a custom sort predicate. Since the values going into it are of "user-defined" type, I cannot modify their built-in comparison predicate. Is there a way to do something like: h = heapq.heapify([...], key=my_lt_pred) h = heapq.heappush(h, key=my_lt_pred) Or even better, I could wrap the heapq functions in my own container so I don't need to keep passing the predicate.

Output: According to the heapq documentation, the way to customize the heap order is to have each element on the heap to be a tuple, with the first tuple element being one that accepts normal Python comparisons. The functions in the heapq module are a bit cumbersome (since they are not object-oriented), and always require our heap object (a heapified list) to be explicitly passed as the first parameter. We can kill two birds with one stone by creating a very simple wrapper class that will allow us to specify a key function, and present the heap as an object. The class below keeps an internal list, where each element is a tuple, the first member of which is a key, calculated at element insertion time using the key parameter, passed at Heap instantiation: # -*- coding: utf-8 -*- import heapq class MyHeap(object): def __init__(self, initial=None, key=lambda x:x): self.key = key self.index = 0 if initial: self._data = [(key(item), i, item) for i, item in enumerate(initial)] self.index = len(self._data) heapq.heapify(self._data) else: self._data = [] def push(self, item): heapq.heappush(self._data, (self.key(item), self.index, item)) self.index += 1 def pop(self): return heapq.heappop(self._data)[2] (The extra self.index part is to avoid clashes when the evaluated key value is a draw and the stored value is not directly comparable - otherwise heapq could fail with TypeError)


Instruction:
String formatting named parameters?

I know it's a really simple question, but I have no idea how to google it. how can I do print '%s' % (my_url) So that my_url is used twice? I assume I have to "name" the %s and then use a dict in the params, but I'm not sure of the proper syntax? just FYI, I'm aware I can just use my_url twice in the params, but that's not the point :)

Output: Solution in Python 3.6+ Python 3.6 introduces literal string formatting, so that you can format the named parameters without any repeating any of your named parameters outside the string: print(f'{my_url:s}') This will evaluate my_url, so if it's not defined you will get a NameError. In fact, instead of my_url, you can write an arbitrary Python expression, as long as it evaluates to a string (because of the :s formatting code). If you want a string representation for the result of an expression that might not be a string, replace :s by !s, just like with regular, pre-literal string formatting. For details on literal string formatting, see PEP 498, where it was first introduced, and PEP 701, where permitted syntax was extended (starting with Python 3.12).


Instruction:
How can I pass data from Flask to JavaScript in a template?

My app makes a call to an API that returns a dictionary. I want to pass information from this dict to JavaScript in the view. I am using the Google Maps API in the JS, specifically, so I'd like to pass it a list of tuples with the long/lat information. I know that render_template will pass these variables to the view so they can be used in HTML, but how could I pass them to JavaScript in the template? from flask import Flask from flask import render_template app = Flask(__name__) import foo_api api = foo_api.API('API KEY') @app.route('/') def get_data(): events = api.call(get_event, arg0, arg1) geocode = event['latitude'], event['longitude'] return render_template('get_data.html', geocode=geocode)

Output: You can use {{ variable }} anywhere in your template, not just in the HTML part. So this should work: <html> <head> <script> var someJavaScriptVar = '{{ geocode[1] }}'; </script> </head> <body> <p>Hello World</p> <button onclick="alert('Geocode: {{ geocode[0] }} ' + someJavaScriptVar)" /> </body> </html> Think of it as a two-stage process: First, Jinja (the template engine Flask uses) generates your text output. This gets sent to the user who executes the JavaScript he sees. If you want your Flask variable to be available in JavaScript as an array, you have to generate an array definition in your output: <html> <head> <script> var myGeocode = ['{{ geocode[0] }}', '{{ geocode[1] }}']; </script> </head> <body> <p>Hello World</p> <button onclick="alert('Geocode: ' + myGeocode[0] + ' ' + myGeocode[1])" /> </body> </html> Jinja also offers more advanced constructs from Python, so you can shorten it to: <html> <head> <script> var myGeocode = [{{ ', '.join(geocode) }}]; </script> </head> <body> <p>Hello World</p> <button onclick="alert('Geocode: ' + myGeocode[0] + ' ' + myGeocode[1])" /> </body> </html> You can also use for loops, if statements and many more, see the Jinja2 documentation for more. Also, have a look at Ford's answer who points out the tojson filter which is an addition to Jinja2's standard set of filters. Edit Nov 2018: tojson is now included in Jinja2's standard set of filters.


Instruction:
Does pandas iterrows have performance issues?

I have noticed very poor performance when using iterrows from pandas. Is it specific to iterrows and should this function be avoided for data of a certain size (I'm working with 2-3 million rows)? This discussion on GitHub led me to believe it is caused when mixing dtypes in the dataframe, however the simple example below shows it is there even when using one dtype (float64). This takes 36 seconds on my machine: import pandas as pd import numpy as np import time s1 = np.random.randn(2000000) s2 = np.random.randn(2000000) dfa = pd.DataFrame({'s1': s1, 's2': s2}) start = time.time() i=0 for rowindex, row in dfa.iterrows(): i+=1 end = time.time() print end - start Why are vectorized operations like apply so much quicker? I imagine there must be some row by row iteration going on there too. I cannot figure out how to not use iterrows in my case (this I'll save for a future question). Therefore I would appreciate hearing if you have consistently been able to avoid this iteration. I'm making calculations based on data in separate dataframes. A simplified version of what I want to run: import pandas as pd import numpy as np #%% Create the original tables t1 = {'letter':['a','b'], 'number1':[50,-10]} t2 = {'letter':['a','a','b','b'], 'number2':[0.2,0.5,0.1,0.4]} table1 = pd.DataFrame(t1) table2 = pd.DataFrame(t2) #%% Create the body of the new table table3 = pd.DataFrame(np.nan, columns=['letter','number2'], index=[0]) #%% Iterate through filtering relevant data, optimizing, returning info for row_index, row in table1.iterrows(): t2info = table2[table2.letter == row['letter']].reset_index() table3.ix[row_index,] = optimize(t2info,row['number1']) #%% Define optimization def optimize(t2info, t1info): calculation = [] for index, r in t2info.iterrows(): calculation.append(r['number2']*t1info) maxrow = calculation.index(max(calculation)) return t2info.ix[maxrow]

Output: Generally, iterrows should only be used in very, very specific cases. This is the general order of precedence for performance of various operations: vectorization using a custom Cython routine apply reductions that can be performed in Cython iteration in Python space itertuples iterrows updating an empty frame (e.g., using loc one-row-at-a-time) Using a custom Cython routine is usually too complicated, so let's skip that for now. Vectorization is always, always the first and best choice. However, there is a small set of cases (usually involving a recurrence) which cannot be vectorized in obvious ways. Furthermore, on a smallish DataFrame, it may be faster to use other methods. apply usually can be handled by an iterator in Cython space. This is handled internally by pandas, though it depends on what is going on inside the apply expression. For example, df.apply(lambda x: np.sum(x)) will be executed pretty swiftly, though of course, df.sum(1) is even better. However something like df.apply(lambda x: x['b'] + 1) will be executed in Python space, and consequently is much slower. itertuples does not box the data into a Series. It just returns the data in the form of tuples. iterrows does box the data into a Series. Unless you really need this, use another method. Updating an empty frame a-single-row-at-a-time. I have seen this method used WAY too much. It is by far the slowest. It is probably common place (and reasonably fast for some Python structures), but a DataFrame does a fair number of checks on indexing, so this will always be very slow to update a row at a time. Much better to create new structures and concat.


Instruction:
How should I read a file line-by-line in Python?

In pre-historic times (Python 1.4) we did: fp = open('filename.txt') while 1: line = fp.readline() if not line: break print(line) after Python 2.1, we did: for line in open('filename.txt').xreadlines(): print(line) before we got the convenient iterator protocol in Python 2.3, and could do: for line in open('filename.txt'): print(line) I've seen some examples using the more verbose: with open('filename.txt') as fp: for line in fp: print(line) Is this the preferred method going forward? [edit] I get that the with statement ensures closing of the file. Why isn't that included in the iterator protocol for file objects?

Output: There is exactly one reason why the following is preferred: with open('filename.txt') as fp: for line in fp: print(line) We are all spoiled by CPython's relatively deterministic reference-counting scheme for garbage collection. Other, hypothetical implementations of Python will not necessarily close the file "quickly enough" without the with block if they use some other scheme to reclaim memory. In such an implementation, you might get a "too many files open" error from the OS if your code opens files faster than the garbage collector calls finalizers on orphaned file handles. The usual workaround is to trigger the GC immediately, but this is a nasty hack and it has to be done by every function that could encounter the error, including those in libraries. What a nightmare. Or you could just use the with block. Bonus Question (Stop reading now if are only interested in the objective aspects of the question.) Why isn't that included in the iterator protocol for file objects? This is a subjective question about API design, so I have a subjective answer in two parts. On a gut level, this feels wrong, because it makes iterator protocol do two separate things—iterate over lines and close the file handle—and it's often a bad idea to make a simple-looking function do two actions. In this case, it feels especially bad because iterators relate in a quasi-functional, value-based way to the contents of a file, but managing file handles is a completely separate task. Squashing both, invisibly, into one action, is surprising to humans who read the code and makes it more difficult to reason about program behavior. Other languages have essentially come to the same conclusion. Haskell briefly flirted with so-called "lazy IO" which allows you to iterate over a file and have it automatically closed when you get to the end of the stream, but it's almost universally discouraged to use lazy IO in Haskell these days, and Haskell users have mostly moved to more explicit resource management like Conduit which behaves more like the with block in Python. On a technical level, there are some things you may want to do with a file handle in Python which would not work as well if iteration closed the file handle. For example, suppose I need to iterate over the file twice: with open('filename.txt') as fp: for line in fp: ... fp.seek(0) for line in fp: ... While this is a less common use case, consider the fact that I might have just added the three lines of code at the bottom to an existing code base which originally had the top three lines. If iteration closed the file, I wouldn't be able to do that. So keeping iteration and resource management separate makes it easier to compose chunks of code into a larger, working Python program. Composability is one of the most important usability features of a language or API.


Instruction:
SQLAlchemy: how to filter date field?

Here is model: class User(Base): ... birthday = Column(Date, index=True) #in database it's like '1987-01-17' ... I want to filter between two dates, for example to choose all users in interval 18-30 years. How to implement it with SQLAlchemy? I think of: query = DBSession.query(User).filter( and_(User.birthday >= '1988-01-17', User.birthday <= '1985-01-17') ) # means age >= 24 and age <= 27 I know this is not correct, but how to do correct?

Output: In fact, your query is right except for the typo: your filter is excluding all records: you should change the <= for >= and vice versa: qry = DBSession.query(User).filter( and_(User.birthday <= '1988-01-17', User.birthday >= '1985-01-17')) # or same: qry = DBSession.query(User).filter(User.birthday <= '1988-01-17').\ filter(User.birthday >= '1985-01-17') Also you can use between: qry = DBSession.query(User).filter(User.birthday.between('1985-01-17', '1988-01-17'))


Instruction:
how to split an iterable in constant-size chunks

I am surprised I could not find a "batch" function that would take as input an iterable and return an iterable of iterables. For example: for i in batch(range(0,10), 1): print i [0] [1] ... [9] or: for i in batch(range(0,10), 3): print i [0,1,2] [3,4,5] [6,7,8] [9] Now, I wrote what I thought was a pretty simple generator: def batch(iterable, n = 1): current_batch = [] for item in iterable: current_batch.append(item) if len(current_batch) == n: yield current_batch current_batch = [] if current_batch: yield current_batch But the above does not give me what I would have expected: for x in batch(range(0,10),3): print x [0] [0, 1] [0, 1, 2] [3] [3, 4] [3, 4, 5] [6] [6, 7] [6, 7, 8] [9] So, I have missed something and this probably shows my complete lack of understanding of python generators. Anyone would care to point me in the right direction ? [Edit: I eventually realized that the above behavior happens only when I run this within ipython rather than python itself]

Output: This is probably more efficient (faster) def batch(iterable, n=1): l = len(iterable) for ndx in range(0, l, n): yield iterable[ndx:min(ndx + n, l)] for x in batch(range(0, 10), 3): print x Example using list data = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10] # list of data for x in batch(data, 3): print(x) # Output [0, 1, 2] [3, 4, 5] [6, 7, 8] [9, 10] It avoids building new lists.


Instruction:
Why do circular imports seemingly work further up in the call stack but then raise an ImportError further down?

I'm getting this error Traceback (most recent call last): File "/Users/alex/dev/runswift/utils/sim2014/simulator.py", line 3, in <module> from world import World File "/Users/alex/dev/runswift/utils/sim2014/world.py", line 2, in <module> from entities.field import Field File "/Users/alex/dev/runswift/utils/sim2014/entities/field.py", line 2, in <module> from entities.goal import Goal File "/Users/alex/dev/runswift/utils/sim2014/entities/goal.py", line 2, in <module> from entities.post import Post File "/Users/alex/dev/runswift/utils/sim2014/entities/post.py", line 4, in <module> from physics import PostBody File "/Users/alex/dev/runswift/utils/sim2014/physics.py", line 21, in <module> from entities.post import Post ImportError: cannot import name Post and you can see that I use the same import statement further up and it works. Is there some unwritten rule about circular importing? How do I use the same class further down the call stack? See also What happens when using mutual or circular (cyclic) imports in Python? for a general overview of what is allowed and what causes a problem WRT circular imports. See What can I do about "ImportError: Cannot import name X" or "AttributeError: ... (most likely due to a circular import)"? for techniques for resolving and avoiding circular dependencies.

Output: I think the answer by jpmc26, while by no means wrong, comes down too heavily on circular imports. They can work just fine, if you set them up correctly. The easiest way to do so is to use import my_module syntax, rather than from my_module import some_object. The former will almost always work, even if my_module included imports us back. The latter only works if my_object is already defined in my_module, which in a circular import may not be the case. To be specific to your case: Try changing entities/post.py to do import physics and then refer to physics.PostBody rather than just PostBody directly. Similarly, change physics.py to do import entities.post and then use entities.post.Post rather than just Post.


Instruction:
Create PDF from a list of images

Is there any practical way to create a PDF from a list of images files, using Python? In Perl I know that module. With it I can create a PDF in just 3 lines: use PDF::FromImage; ... my $pdf = PDF::FromImage->new; $pdf->load_images(@allPagesDir); $pdf->write_file($bookName . '.pdf'); I need to do something very similar to this, but in Python. I know the pyPdf module, but I would like something simple.

Output: Install fpdf2 for Python: pip install fpdf2 Now you can use the same logic: from fpdf import FPDF pdf = FPDF() # imagelist is the list with all image filenames for image in imagelist: pdf.add_page() pdf.image(image,x,y,w,h) pdf.output("yourfile.pdf", "F") You can find more info at the tutorial page or the official documentation.


Instruction:
How do lexical closures work?

While I was investigating a problem I had with lexical closures in Javascript code, I came along this problem in Python: flist = [] for i in xrange(3): def func(x): return x * i flist.append(func) for f in flist: print f(2) Note that this example mindfully avoids lambda. It prints "4 4 4", which is surprising. I'd expect "0 2 4". This equivalent Perl code does it right: my @flist = (); foreach my $i (0 .. 2) { push(@flist, sub {$i * $_[0]}); } foreach my $f (@flist) { print $f->(2), "\n"; } "0 2 4" is printed. Can you please explain the difference ? Update: The problem is not with i being global. This displays the same behavior: flist = [] def outer(): for i in xrange(3): def inner(x): return x * i flist.append(inner) outer() #~ print i # commented because it causes an error for f in flist: print f(2) As the commented line shows, i is unknown at that point. Still, it prints "4 4 4".

Output: Python is actually behaving as defined. Three separate functions are created, but they each have the closure of the environment they're defined in - in this case, the global environment (or the outer function's environment if the loop is placed inside another function). This is exactly the problem, though - in this environment, i is modified, and the closures all refer to the same i. Here is the best solution I can come up with - create a function creater and invoke that instead. This will force different environments for each of the functions created, with a different i in each one. flist = [] for i in xrange(3): def funcC(j): def func(x): return x * j return func flist.append(funcC(i)) for f in flist: print f(2) This is what happens when you mix side effects and functional programming.


Instruction:
Extract first and last row of a dataframe in pandas

How can I extract the first and last rows of a given dataframe as a new dataframe in pandas? I've tried to use iloc to select the desired rows and then concat as in: df=pd.DataFrame({'a':range(1,5), 'b':['a','b','c','d']}) pd.concat([df.iloc[0,:], df.iloc[-1,:]]) but this does not produce a pandas dataframe: a 1 b a a 4 b d dtype: object

Output: Some of the other answers duplicate the first row if the frame only contains a single row. If that's a concern df[0::len(df)-1 if len(df) > 1 else 1] works even for single row-dataframes. Example: For the following dataframe this will not create a duplicate: df = pd.DataFrame({'a': [1], 'b':['a']}) df2 = df[0::len(df)-1 if len(df) > 1 else 1] print df2 a b 0 1 a whereas this does: df3 = df.iloc[[0, -1]] print df3 a b 0 1 a 0 1 a because the single row is the first AND last row at the same time.


Instruction:
Why does random.shuffle return None?

Why is random.shuffle returning None in Python? >>> x = ['foo','bar','black','sheep'] >>> from random import shuffle >>> print shuffle(x) None How do I get the shuffled value instead of None?

Output: random.shuffle() changes the x list in place. Python API methods that alter a structure in-place generally return None, not the modified data structure. >>> x = ['foo', 'bar', 'black', 'sheep'] >>> random.shuffle(x) >>> x ['black', 'bar', 'sheep', 'foo'] If you wanted to create a new randomly-shuffled list based on an existing one, where the existing list is kept in order, you could use random.sample() with the full length of the input: random.sample(x, len(x)) You could also use sorted() with random.random() for a sorting key: shuffled = sorted(x, key=lambda k: random.random()) but this invokes sorting (an O(N log N) operation), while sampling to the input length only takes O(N) operations (the same process as random.shuffle() is used, swapping out random values from a shrinking pool). Demo: >>> import random >>> x = ['foo', 'bar', 'black', 'sheep'] >>> random.sample(x, len(x)) ['bar', 'sheep', 'black', 'foo'] >>> sorted(x, key=lambda k: random.random()) ['sheep', 'foo', 'black', 'bar'] >>> x ['foo', 'bar', 'black', 'sheep']


Instruction:
What does this tensorflow message mean? Any side effect? Was the installation successful?

I just installed tensorflow v2.3 on anaconda python. I tried to test out the installation using the python command below; $ python -c "import tensorflow as tf; x = [[2.]]; print('tensorflow version', tf.__version__); print('hello, {}'.format(tf.matmul(x, x)))" I got the following message; 2020-12-15 07:59:12.411952: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations: AVX AVX2 To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. hello, [[4.]] From the message, it seems that the installation was installed successfully. But what does This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations: AVX AVX2 mean exactly? Am I using a tensorflow version with some limited features? Any side effects? I am using Windows 10.

Output: An important part of Tensorflow is that it is supposed to be fast. With a suitable installation, it works with CPUs, GPUs, or TPUs. Part of going fast means that it uses different code depending on your hardware. Some CPUs support operations that other CPUs do not, such as vectorized addition (adding multiple variables at once). Tensorflow is simply telling you that the version you have installed can use the AVX and AVX2 operations and is set to do so by default in certain situations (say inside a forward or back-prop matrix multiply), which can speed things up. This is not an error, it is just telling you that it can and will take advantage of your CPU to get that extra speed out. Note: AVX stands for Advanced Vector Extensions.


Instruction:
How to convert list of tuples to multiple lists?

Suppose I have a list of tuples and I want to convert to multiple lists. For example, the list of tuples is [(1,2),(3,4),(5,6),] Is there any built-in function in Python that convert it to: [1,3,5],[2,4,6] This can be a simple program. But I am just curious about the existence of such built-in function in Python.

Output: The built-in function zip() will almost do what you want: >>> list(zip(*[(1, 2), (3, 4), (5, 6)])) [(1, 3, 5), (2, 4, 6)] This will give you a list of tuples. If you want to go further and arrive at a list of lists: >>> list(map(list, zip(*[(1, 2), (3, 4), (5, 6)]))) [[1, 3, 5], [2, 4, 6]]


Instruction:
What algorithm does python's sorted() use?

In Python 2.7, how does Python's built-in sorted function work - what algorithm does it use?

Output: Older versions of Python (2.3 - 3.10) used an algorithm called Timsort: Timsort is a hybrid sorting algorithm, derived from merge sort and insertion sort, designed to perform well on many kinds of real-world data. It was invented by Tim Peters in 2002 for use in the Python programming language. The algorithm finds subsets of the data that are already ordered, and uses the subsets to sort the data more efficiently. This is done by merging an identified subset, called a run, with existing runs until certain criteria are fulfilled. Timsort was Python's standard sorting algorithm from version 2.3 to version 3.10. It is now also used to sort arrays in Java SE 7, and on the Android platform. Since 3.11, Python uses Powersort, which was designed by Ian Munro and Sebastian Wild. It is an improved nearly-optimal mergesort that adapts to existing runs of sorted data.


Instruction:
Multiple Python versions on the same machine?

Is there official documentation on the Python website somewhere, on how to install and run multiple versions of Python on the same machine on Linux? I can find gazillions of blog posts and answers, but I want to know if there is a "standard" official way of doing this? Or is this all dependent on OS?

Output: I think it is totally independent. Just install them, then you have the commands e.g. /usr/bin/python2.5 and /usr/bin/python2.6. Link /usr/bin/python to the one you want to use as default. All the libraries are in separate folders (named after the version) anyway. If you want to compile the versions manually, this is from the readme file of the Python source code: Installing multiple versions On Unix and Mac systems if you intend to install multiple versions of Python using the same installation prefix (--prefix argument to the configure script) you must take care that your primary python executable is not overwritten by the installation of a different version. All files and directories installed using make altinstall contain the major and minor version and can thus live side-by-side. make install also creates ${prefix}/bin/python3 which refers to ${prefix}/bin/pythonX.Y. If you intend to install multiple versions using the same prefix you must decide which version (if any) is your "primary" version. Install that version using make install. Install all other versions using make altinstall. For example, if you want to install Python 2.5, 2.6 and 3.0 with 2.6 being the primary version, you would execute make install in your 2.6 build directory and make altinstall in the others.


Instruction:
Properties file in python (similar to Java Properties)

Given the following format (.properties or .ini): propertyName1=propertyValue1 propertyName2=propertyValue2 ... propertyNameN=propertyValueN For Java there is the Properties class that offers functionality to parse / interact with the above format. Is there something similar in python's standard library (2.x) ? If not, what other alternatives do I have ?

Output: For .ini files there is the configparser module that provides a format compatible with .ini files. Anyway there's nothing available for parsing complete .properties files, when I have to do that I simply use jython (I'm talking about scripting).


Instruction:
seek() function?

Please excuse my confusion here but I have read the documentation regarding the seek() function in Python (after having to use it). Although it helped me I am still a bit confused on the actual meaning of what it does, any explanations are much appreciated. Thank you.

Output: Regarding seek() there's not too much to worry about. First of all, it is useful when operating over an open file. It's important to note that its syntax is as follows: fp.seek(offset, from_what) where fp is the file pointer you're working with; offset means how many positions you will move; from_what defines your point of reference: 0: means your reference point is the beginning of the file 1: means your reference point is the current file position 2: means your reference point is the end of the file if omitted, from_what defaults to 0. Never forget that when managing files, there'll always be a position inside that file where you are currently working on. When just open, that position is the beginning of the file, but as you work with it, you may advance. seek will be useful to you when you need to walk along that open file, just as a path you are traveling into.


Instruction:
Conditional/ternary operator for expressions in Python

Many languages have a conditional (AKA ternary) operator. This allows you to make terse choices between two values based on a condition, which makes expressions, including assignments, concise. My code has conditional assignments: if condition: var = something else: var = something_else In C it'd be: var = condition ? something : something_else; In Python, is there a trick you can use to get the assignment onto a single line to approximate the advantages of the conditional operator?

Output: Python has such an operator: variable = something if condition else something_else Alternatively, although not recommended (see karadoc's comment): variable = (condition and something) or something_else


Instruction:
Deep-Learning Nan loss reasons

What would cause a Convolutional Neural Network to diverge? Specifics: I am using Tensorflow's iris_training model with some of my own data and keep getting ERROR:tensorflow:Model diverged with loss = NaN. Traceback... tensorflow.contrib.learn.python.learn.monitors.NanLossDuringTrainingError: NaN loss during training. Traceback originated with line: tf.contrib.learn.DNNClassifier(feature_columns=feature_columns, hidden_units=[300, 300, 300], #optimizer=tf.train.ProximalAdagradOptimizer(learning_rate=0.001, l1_regularization_strength=0.00001), n_classes=11, model_dir="/tmp/iris_model") I've tried adjusting the optimizer, using a zero for learning rate, and using no optimizer.

Output: There are lots of things I have seen make a model diverge. Too high of a learning rate. You can often tell if this is the case if the loss begins to increase and then diverges to infinity. I am not to familiar with the DNNClassifier but I am guessing it uses the categorical cross entropy cost function. This involves taking the log of the prediction which diverges as the prediction approaches zero. That is why people usually add a small epsilon value to the prediction to prevent this divergence. I am guessing the DNNClassifier probably does this or uses the tensorflow opp for it. Probably not the issue. Other numerical stability issues can exist such as division by zero where adding the epsilon can help. Another less obvious one if the square root whose derivative can diverge if not properly simplified when dealing with finite precision numbers. Yet again I doubt this is the issue in the case of the DNNClassifier. You may have an issue with the input data. Try calling assert not np.any(np.isnan(x)) on the input data to make sure you are not introducing the nan. Also make sure all of the target values are valid. Finally, make sure the data is properly normalized. You probably want to have the pixels in the range [-1, 1] and not [0, 255]. The labels must be in the domain of the loss function, so if using a logarithmic-based loss function all labels must be non-negative (as noted by evan pu and the comments below).


Instruction:
Django - how to create a file and save it to a model's FileField?

Here's my model. What I want to do is generate a new file and overwrite the existing one whenever a model instance is saved: class Kitten(models.Model): claw_size = ... license_file = models.FileField(blank=True, upload_to='license') def save(self, *args, **kwargs): #Generate a new license file overwriting any previous version #and update file path self.license_file = ??? super(Request,self).save(*args, **kwargs) I see lots of documentation about how to upload a file. But how do I generate a file, assign it to a model field and have Django store it in the right place?

Output: You want to have a look at FileField and FieldFile in the Django docs, and especially FieldFile.save(). Basically, a field declared as a FileField, when accessed, gives you an instance of class FieldFile, which gives you several methods to interact with the underlying file. So, what you need to do is: self.license_file.save(new_name, new_contents) where new_name is the filename you wish assigned and new_contents is the content of the file. Note that new_contents must be an instance of either django.core.files.File or django.core.files.base.ContentFile (see given links to manual for the details). The two choices boil down to: from django.core.files.base import ContentFile, File # Using File with open('/path/to/file') as f: self.license_file.save(new_name, File(f)) # Using ContentFile self.license_file.save(new_name, ContentFile('A string with the file content'))


Instruction:
What is the difference between NaN and None?

I am reading two columns of a csv file using pandas readcsv() and then assigning the values to a dictionary. The columns contain strings of numbers and letters. Occasionally there are cases where a cell is empty. In my opinion, the value read to that dictionary entry should be None but instead nan is assigned. Surely None is more descriptive of an empty cell as it has a null value, whereas nan just says that the value read is not a number. Is my understanding correct, what IS the difference between None and nan? Why is nan assigned instead of None? Also, my dictionary check for any empty cells has been using numpy.isnan(): for k, v in my_dict.iteritems(): if np.isnan(v): But this gives me an error saying that I cannot use this check for v. I guess it is because an integer or float variable, not a string is meant to be used. If this is true, how can I check v for an "empty cell"/nan case?

Output: NaN is used as a placeholder for missing data consistently in pandas, consistency is good. I usually read/translate NaN as "missing". Also see the 'working with missing data' section in the docs. Wes writes in the docs 'choice of NA-representation': After years of production use [NaN] has proven, at least in my opinion, to be the best decision given the state of affairs in NumPy and Python in general. The special value NaN (Not-A-Number) is used everywhere as the NA value, and there are API functions isna and notna which can be used across the dtypes to detect NA values. ... Thus, I have chosen the Pythonic “practicality beats purity” approach and traded integer NA capability for a much simpler approach of using a special value in float and object arrays to denote NA, and promoting integer arrays to floating when NAs must be introduced. Note: the "gotcha" that integer Series containing missing data are upcast to floats. In my opinion the main reason to use NaN (over None) is that it can be stored with numpy's float64 dtype, rather than the less efficient object dtype, see NA type promotions. # without forcing dtype it changes None to NaN! s_bad = pd.Series([1, None], dtype=object) s_good = pd.Series([1, np.nan]) In [13]: s_bad.dtype Out[13]: dtype('O') In [14]: s_good.dtype Out[14]: dtype('float64') Jeff comments (below) on this: np.nan allows for vectorized operations; its a float value, while None, by definition, forces object type, which basically disables all efficiency in numpy. So repeat 3 times fast: object==bad, float==good Saying that, many operations may still work just as well with None vs NaN (but perhaps are not supported i.e. they may sometimes give surprising results): In [15]: s_bad.sum() Out[15]: 1 In [16]: s_good.sum() Out[16]: 1.0 To answer the second question: You should be using isna and notna to test for missing data (NaN).


Instruction:
Return datetime object of previous month

If only timedelta had a month argument in it's constructor. So what's the simplest way to do this? EDIT: I wasn't thinking too hard about this as was pointed out below. Really what I wanted was any day in the last month because eventually I'm going to grab the year and month only. So given a datetime object, what's the simplest way to return any datetime object that falls in the previous month?

Output: Try this: def monthdelta(date, delta): m, y = (date.month+delta) % 12, date.year + ((date.month)+delta-1) // 12 if not m: m = 12 d = min(date.day, [31, 29 if y%4==0 and (not y%100==0 or y%400 == 0) else 28, 31,30,31,30,31,31,30,31,30,31][m-1]) return date.replace(day=d,month=m, year=y) >>> for m in range(-12, 12): print(monthdelta(datetime.now(), m)) 2009-08-06 16:12:27.823000 2009-09-06 16:12:27.855000 2009-10-06 16:12:27.870000 2009-11-06 16:12:27.870000 2009-12-06 16:12:27.870000 2010-01-06 16:12:27.870000 2010-02-06 16:12:27.870000 2010-03-06 16:12:27.886000 2010-04-06 16:12:27.886000 2010-05-06 16:12:27.886000 2010-06-06 16:12:27.886000 2010-07-06 16:12:27.886000 2010-08-06 16:12:27.901000 2010-09-06 16:12:27.901000 2010-10-06 16:12:27.901000 2010-11-06 16:12:27.901000 2010-12-06 16:12:27.901000 2011-01-06 16:12:27.917000 2011-02-06 16:12:27.917000 2011-03-06 16:12:27.917000 2011-04-06 16:12:27.917000 2011-05-06 16:12:27.917000 2011-06-06 16:12:27.933000 2011-07-06 16:12:27.933000 >>> monthdelta(datetime(2010,3,30), -1) datetime.datetime(2010, 2, 28, 0, 0) >>> monthdelta(datetime(2008,3,30), -1) datetime.datetime(2008, 2, 29, 0, 0) Edit Corrected to handle the day as well. Edit See also the answer from puzzlement which points out a simpler calculation for d: d = min(date.day, calendar.monthrange(y, m)[1])


Instruction:
Stopping Python using Ctrl + C

I have a Python script that uses threads and makes lots of HTTP requests. I think, while a HTTP request (using urllib2) is reading, it's blocking and not responding to Ctrl + C to stop the program. Is there a way around this?

Output: On Windows, the only sure way is to use Ctrl + Break. It stops every Python script instantly! (Note that on some keyboards, "Break" is labeled as "Pause".)


Instruction:
Specify date format for Python argparse input arguments

I have a Python script that requires some command line inputs and I am using argparse for parsing them. I found the documentation a bit confusing and couldn't find a way to check for a format in the input parameters. What I mean by checking format is explained with this example script: parser.add_argument('-s', "--startdate", help="The Start Date - format YYYY-MM-DD ", required=True) parser.add_argument('-e', "--enddate", help="The End Date format YYYY-MM-DD (Inclusive)", required=True) parser.add_argument('-a', "--accountid", type=int, help='Account ID for the account for which data is required (Default: 570)') parser.add_argument('-o', "--outputpath", help='Directory where output needs to be stored (Default: ' + os.path.dirname(os.path.abspath(__file__))) I need to check for option -s and -e that the input by the user is in the format YYYY-MM-DD. Is there an option in argparse that I do not know of which accomplishes this?

Output: Per the documentation: The type keyword argument of add_argument() allows any necessary type-checking and type conversions to be performed ... The argument to type can be any callable that accepts a single string. You could do something like: import argparse import datetime def valid_date(s: str) -> datetime.datetime: try: return datetime.datetime.strptime(s, "%Y-%m-%d") except ValueError: raise argparse.ArgumentTypeError(f"not a valid date: {s!r}") Then use that as type: parser.add_argument( "-s", "--startdate", help="The Start Date - format YYYY-MM-DD", required=True, type=valid_date ) If the user supplies an invalid value, the feedback will look like: error: argument -s/--startdate: not a valid date: 'foo'


Instruction:
How to send mail with To, CC and BCC?

I need to populate a few hundred email boxes with various messages for testing purposes and I was going to use smtplib for that. I need to send messages not only To specific mailboxes, but CC and BCC them as well. How do I do that?

Output: Email headers don't matter to the SMTP server. Just add CC and BCC recipients to to_addrs when sending emails. For CC, add them to the CC header. toaddr = 'buffy@sunnydale.k12.ca.us' cc = ['alexander@sunydale.k12.ca.us','willow@sunnydale.k12.ca.us'] bcc = ['chairman@slayerscouncil.uk'] fromaddr = 'giles@sunnydale.k12.ca.us' message_subject = "disturbance in sector 7" message_text = "Three are dead in an attack in the sewers below sector 7." message = "From: %s\r\n" % fromaddr + "To: %s\r\n" % toaddr + "CC: %s\r\n" % ",".join(cc) + "Subject: %s\r\n" % message_subject + "\r\n" + message_text toaddrs = [toaddr] + cc + bcc server = smtplib.SMTP('smtp.sunnydale.k12.ca.us') server.set_debuglevel(1) server.sendmail(fromaddr, toaddrs, message) server.quit()


Instruction:
How do I find the maximum (larger, greater) of 2 numbers?

I have two variables value and run: value = -9999 run = problem.getscore() How can I find out which one is greater, and get the greater value? See also Find the greatest (largest, maximum) number in a list of numbers - those approaches work (and are shown here), but two numbers can also be compared directly.

Output: Use the builtin function max. Example: max(2, 4) returns 4. Just for giggles, there's a min as well...should you need it. :P


Instruction:
How to use hex() without 0x in Python?

The hex() function in python, puts the leading characters 0x in front of the number. Is there anyway to tell it NOT to put them? So 0xfa230 will be fa230. The code is import fileinput f = open('hexa', 'w') for line in fileinput.input(['pattern0.txt']): f.write(hex(int(line))) f.write('\n')

Output: (Recommended) Python 3 f-strings: Answered by @GringoSuave >>> i = 3735928559 >>> f'{i:x}' 'deadbeef' Alternatives: format builtin function (good for single values only) >>> format(3735928559, 'x') 'deadbeef' And sometimes we still may need to use str.format formatting in certain situations @Eumiro (Though I would still recommend f-strings in most situations) >>> '{:x}'.format(3735928559) 'deadbeef' (Legacy) f-strings should solve all of your needs, but printf-style formatting is what we used to do @msvalkon >>> '%x' % 3735928559 'deadbeef' Without string formatting @jsbueno >>> i = 3735928559 >>> i.to_bytes(4, "big").hex() 'deadbeef' Hacky Answers (avoid) hex(i)[2:] @GuillaumeLemaître >>> i = 3735928559 >>> hex(i)[2:] 'deadbeef' This relies on string slicing instead of using a function / method made specifically for formatting as hex. This is why it may give unexpected output for negative numbers: >>> i = -3735928559 >>> hex(i)[2:] 'xdeadbeef' >>> f'{i:x}' '-deadbeef'


Instruction:
How can I remove the ANSI escape sequences from a string in python

Here is a snippet that includes my string. 'ls\r\n\x1b[00m\x1b[01;31mexamplefile.zip\x1b[00m\r\n\x1b[01;31m' The string was returned from an SSH command that I executed. I can't use the string in its current state because it contains ANSI standardized escape sequences. How can I programmatically remove the escape sequences so that the only part of the string remaining is 'examplefile.zip'.

Output: Delete them with a regular expression: import re # 7-bit C1 ANSI sequences ansi_escape = re.compile(r''' \x1B # ESC (?: # 7-bit C1 Fe (except CSI) [@-Z\\-_] | # or [ for CSI, followed by a control sequence \[ [0-?]* # Parameter bytes [ -/]* # Intermediate bytes [@-~] # Final byte ) ''', re.VERBOSE) result = ansi_escape.sub('', sometext) or, without the VERBOSE flag, in condensed form: ansi_escape = re.compile(r'\x1B(?:[@-Z\\-_]|\[[0-?]*[ -/]*[@-~])') result = ansi_escape.sub('', sometext) Demo: >>> import re >>> ansi_escape = re.compile(r'\x1B(?:[@-Z\\-_]|\[[0-?]*[ -/]*[@-~])') >>> sometext = 'ls\r\n\x1b[00m\x1b[01;31mexamplefile.zip\x1b[00m\r\n\x1b[01;31m' >>> ansi_escape.sub('', sometext) 'ls\r\nexamplefile.zip\r\n' The above regular expression covers all 7-bit ANSI C1 escape sequences, but not the 8-bit C1 escape sequence openers. The latter are never used in today's UTF-8 world where the same range of bytes have a different meaning. If you do need to cover the 8-bit codes too (and are then, presumably, working with bytes values) then the regular expression becomes a bytes pattern like this: # 7-bit and 8-bit C1 ANSI sequences ansi_escape_8bit = re.compile(br''' (?: # either 7-bit C1, two bytes, ESC Fe (omitting CSI) \x1B [@-Z\\-_] | # or a single 8-bit byte Fe (omitting CSI) [\x80-\x9A\x9C-\x9F] | # or CSI + control codes (?: # 7-bit CSI, ESC [ \x1B\[ | # 8-bit CSI, 9B \x9B ) [0-?]* # Parameter bytes [ -/]* # Intermediate bytes [@-~] # Final byte ) ''', re.VERBOSE) result = ansi_escape_8bit.sub(b'', somebytesvalue) which can be condensed down to # 7-bit and 8-bit C1 ANSI sequences ansi_escape_8bit = re.compile( br'(?:\x1B[@-Z\\-_]|[\x80-\x9A\x9C-\x9F]|(?:\x1B\[|\x9B)[0-?]*[ -/]*[@-~])' ) result = ansi_escape_8bit.sub(b'', somebytesvalue) For more information, see: the ANSI escape codes overview on Wikipedia ECMA-48 standard, 5th edition (especially sections 5.3 and 5.4) The example you gave contains 4 CSI (Control Sequence Introducer) codes, as marked by the \x1B[ or ESC [ opening bytes, and each contains a SGR (Select Graphic Rendition) code, because they each end in m. The parameters (separated by ; semicolons) in between those tell your terminal what graphic rendition attributes to use. So for each \x1B[....m sequence, the 3 codes that are used are: 0 (or 00 in this example): reset, disable all attributes 1 (or 01 in the example): bold 31: red (foreground) However, there is more to ANSI than just CSI SGR codes. With CSI alone you can also control the cursor, clear lines or the whole display, or scroll (provided the terminal supports this of course). And beyond CSI, there are codes to select alternative fonts (SS2 and SS3), to send 'private messages' (think passwords), to communicate with the terminal (DCS), the OS (OSC), or the application itself (APC, a way for applications to piggy-back custom control codes on to the communication stream), and further codes to help define strings (SOS, Start of String, ST String Terminator) or to reset everything back to a base state (RIS). The above regexes cover all of these. Note that the above regex only removes the ANSI C1 codes, however, and not any additional data that those codes may be marking up (such as the strings sent between an OSC opener and the terminating ST code). Removing those would require additional work outside the scope of this answer.


Instruction:
Spark Dataframe distinguish columns with duplicated name

So as I know in Spark Dataframe, that for multiple columns can have the same name as shown in below dataframe snapshot: [ Row(a=107831, f=SparseVector(5, {0: 0.0, 1: 0.0, 2: 0.0, 3: 0.0, 4: 0.0}), a=107831, f=SparseVector(5, {0: 0.0, 1: 0.0, 2: 0.0, 3: 0.0, 4: 0.0})), Row(a=107831, f=SparseVector(5, {0: 0.0, 1: 0.0, 2: 0.0, 3: 0.0, 4: 0.0}), a=125231, f=SparseVector(5, {0: 0.0, 1: 0.0, 2: 0.0047, 3: 0.0, 4: 0.0043})), Row(a=107831, f=SparseVector(5, {0: 0.0, 1: 0.0, 2: 0.0, 3: 0.0, 4: 0.0}), a=145831, f=SparseVector(5, {0: 0.0, 1: 0.2356, 2: 0.0036, 3: 0.0, 4: 0.4132})), Row(a=107831, f=SparseVector(5, {0: 0.0, 1: 0.0, 2: 0.0, 3: 0.0, 4: 0.0}), a=147031, f=SparseVector(5, {0: 0.0, 1: 0.0, 2: 0.0, 3: 0.0, 4: 0.0})), Row(a=107831, f=SparseVector(5, {0: 0.0, 1: 0.0, 2: 0.0, 3: 0.0, 4: 0.0}), a=149231, f=SparseVector(5, {0: 0.0, 1: 0.0032, 2: 0.2451, 3: 0.0, 4: 0.0042})) ] Above result is created by join with a dataframe to itself, you can see there are 4 columns with both two a and f. The problem is is there when I try to do more calculation with the a column, I cant find a way to select the a, I have try df[0] and df.select('a'), both returned me below error mesaage: AnalysisException: Reference 'a' is ambiguous, could be: a#1333L, a#1335L. Is there anyway in Spark API that I can distinguish the columns from the duplicated names again? or maybe some way to let me change the column names?

Output: I would recommend that you change the column names for your join. df1.select(col("a") as "df1_a", col("f") as "df1_f") .join(df2.select(col("a") as "df2_a", col("f") as "df2_f"), col("df1_a" === col("df2_a")) The resulting DataFrame will have schema (df1_a, df1_f, df2_a, df2_f)


Instruction:
How to convert a date string to different format

I need to convert date string "2013-1-25" to string "1/25/13" in python. I looked at the datetime.strptime but still can't find a way for this.

Output: I assume I have import datetime before running each of the lines of code below datetime.datetime.strptime("2013-1-25", '%Y-%m-%d').strftime('%m/%d/%y') prints "01/25/13". If you can't live with the leading zero, try this: dt = datetime.datetime.strptime("2013-1-25", '%Y-%m-%d') print '{0}/{1}/{2:02}'.format(dt.month, dt.day, dt.year % 100) This prints "1/25/13". EDIT: This may not work on every platform: datetime.datetime.strptime("2013-1-25", '%Y-%m-%d').strftime('%m/%d/%y')


Instruction:
How to read file with space separated values in pandas

I try to read the file into pandas. The file has values separated by space, but with different number of spaces I tried: pd.read_csv('file.csv', delimiter=' ') but it doesn't work

Output: Add delim_whitespace=True argument, it's faster than regex. Note however that after pandas 2.2.0 this is deprecated and it's better to use sep="\s+"


Instruction:
How to extend Python Enum?

Is it possible to extend classes created using the new Enum functionality in Python 3.4? How? Simple subclassing doesn't appear to work. An example like from enum import Enum class EventStatus(Enum): success = 0 failure = 1 class BookingStatus(EventStatus): duplicate = 2 unknown = 3 will give an exception like TypeError: Cannot extend enumerations or (in more recent versions) TypeError: BookingStatus: cannot extend enumeration 'EventStatus'. How can I make it so that BookingStatus reuses the enumeration values from EventStatus and adds more?

Output: Subclassing an enumeration is allowed only if the enumeration does not define any members. Allowing subclassing of enums that define members would lead to a violation of some important invariants of types and instances. https://docs.python.org/3/howto/enum.html#restricted-enum-subclassing So no, it's not directly possible.


Instruction:
When do I need to use sqlalchemy back_populates?

When I try SQLAlchemy Relation Example following this guide: Basic Relationship Patterns I have this code #!/usr/bin/env python # encoding: utf-8 from sqlalchemy import create_engine from sqlalchemy import Table, Column, Integer, ForeignKey from sqlalchemy.orm import relationship, sessionmaker from sqlalchemy.ext.declarative import declarative_base engine = create_engine('sqlite:///:memory:', echo=True) Session = sessionmaker(bind=engine) session = Session() Base = declarative_base(bind=engine) class Parent(Base): __tablename__ = 'parent' id = Column(Integer, primary_key=True) children = relationship("Child") class Child(Base): __tablename__ = 'child' id = Column(Integer, primary_key=True) parent_id = Column(Integer, ForeignKey('parent.id')) parent = relationship("Parent") Base.metadata.create_all() p = Parent() session.add(p) session.commit() c = Child(parent_id=p.id) session.add(c) session.commit() print "children: {}".format(p.children[0].id) print "parent: {}".format(c.parent.id) It works well, but in the guide, it says the model should be: class Parent(Base): __tablename__ = 'parent' id = Column(Integer, primary_key=True) children = relationship("Child", back_populates="parent") class Child(Base): __tablename__ = 'child' id = Column(Integer, primary_key=True) parent_id = Column(Integer, ForeignKey('parent.id')) parent = relationship("Parent", back_populates="children") Why don't I need back_populates or backref in my example? When should I use one or the other?

Output: If you use backref you don't need to declare the relationship on the second table. class Parent(Base): __tablename__ = 'parent' id = Column(Integer, primary_key=True) children = relationship("Child", backref="parent") class Child(Base): __tablename__ = 'child' id = Column(Integer, primary_key=True) parent_id = Column(Integer, ForeignKey('parent.id')) If you're not using backref, and defining the relationship's separately, then if you don't use back_populates, sqlalchemy won't know to connect the relationships, so that modifying one also modifies the other. So, in your example, where you've defined the relationship's separately, but didn't provide a back_populates argument, modifying one field wouldn't automatically update the other in your transaction. >>> parent = Parent() >>> child = Child() >>> child.parent = parent >>> print(parent.children) [] See how it didn't automatically fill out the children field? Now, if you supply a back_populates argument, sqlalchemy will connect the fields. class Parent(Base): __tablename__ = 'parent' id = Column(Integer, primary_key=True) children = relationship("Child", back_populates="parent") class Child(Base): __tablename__ = 'child' id = Column(Integer, primary_key=True) parent_id = Column(Integer, ForeignKey('parent.id')) parent = relationship("Parent", back_populates="children") So now we get >>> parent = Parent() >>> child = Child() >>> child.parent = parent >>> print(parent.children) [Child(...)] Sqlalchemy knows these two fields are related now, and will update each as the other is updated. It's worth noting that using backref will do this, too. Using back_populates is nice if you want to define the relationships on every class, so it's easy to see all the fields just be glancing at the model class, instead of having to look at other classes that define fields via backref.


Instruction:
Why doesn't calling a string method (such as .replace or .strip) modify (mutate) the string?

I wrote this Python program to do a simple string replacement: X = "hello world" X.replace("hello", "goodbye") After this, the value of X was "hello world". Why didn't it changed to "goodbye world" instead?

Output: This is because strings are immutable in Python. Which means that X.replace("hello","goodbye") returns a copy of X with replacements made. Because of that you need to replace this line: X.replace("hello", "goodbye") with this line: X = X.replace("hello", "goodbye") More broadly, this is true for all Python string methods that change a string's content, e.g. replace,strip,translate,lower/upper,join,... You must assign their output to something if you want to use it and not throw it away, e.g. X = X.strip(' \t') X2 = X.translate(...) Y = X.lower() Z = X.upper() A = X.join(':') B = X.capitalize() C = X.casefold() and so on.


Instruction:
How to create major and minor gridlines with different linestyles

I am currently using matplotlib.pyplot to create graphs and would like to have the major gridlines solid and black and the minor ones either greyed or dashed. In the grid properties, which=both/major/mine, and then color and linestyle are defined simply by linestyle. Is there a way to specify minor linestyle only? The appropriate code I have so far is plt.plot(current, counts, 'rd', markersize=8) plt.yscale('log') plt.grid(b=True, which='both', color='0.65', linestyle='-')

Output: Actually, it is as simple as setting major and minor separately: In [9]: plot([23, 456, 676, 89, 906, 34, 2345]) Out[9]: [<matplotlib.lines.Line2D at 0x6112f90>] In [10]: yscale('log') In [11]: grid(visible=True, which='major', color='b', linestyle='-') In [12]: grid(visible=True, which='minor', color='r', linestyle='--') The gotcha with minor grids is that you have to have minor tick marks turned on too. In the above code this is done by yscale('log'), but it can also be done with plt.minorticks_on(). Note: before matplotlib 3.5, visible parameter was named b


Instruction:
PEP 8, why no spaces around '=' in keyword argument or a default parameter value?

Why does PEP 8 recommend not having spaces around = in a keyword argument or a default parameter value? Is this inconsistent with recommending spaces around every other occurrence of = in Python code? How is: func(1, 2, very_long_variable_name=another_very_long_variable_name) better than: func(1, 2, very_long_variable_name = another_very_long_variable_name) Any links to discussion/explanation by Python's BDFL will be appreciated. Mind, this question is more about kwargs than default values, i just used the phrasing from PEP 8. I'm not soliciting opinions. I'm asking for reasons behind this decision. It's more like asking why would I use { on the same line as if statement in a C program, not whether I should use it or not.

Output: I guess that it is because a keyword argument is essentially different than a variable assignment. For example, there is plenty of code like this: kw1 = some_value kw2 = some_value kw3 = some_value some_func( 1, 2, kw1=kw1, kw2=kw2, kw3=kw3) As you see, it makes complete sense to assign a variable to a keyword argument named exactly the same, so it improves readability to see them without spaces. It is easier to recognize that we are using keyword arguments and not assigning a variable to itself. Also, parameters tend to go in the same line whereas assignments usually are each one in their own line, so saving space is likely to be an important matter there.


Instruction:
How to limit concurrency with Python asyncio?

Let's assume we have a bunch of links to download and each of the link may take a different amount of time to download. And I'm allowed to download using utmost 3 connections only. Now, I want to ensure that I do this efficiently using asyncio. Here's what I'm trying to achieve: At any point in time, try to ensure that I have atleast 3 downloads running. Connection 1: 1---------7---9--- Connection 2: 2---4----6----- Connection 3: 3-----5---8----- The numbers represent the download links, while hyphens represent Waiting for download. Here is the code that I'm using right now from random import randint import asyncio count = 0 async def download(code, permit_download, no_concurrent, downloading_event): global count downloading_event.set() wait_time = randint(1, 3) print('downloading {} will take {} second(s)'.format(code, wait_time)) await asyncio.sleep(wait_time) # I/O, context will switch to main function print('downloaded {}'.format(code)) count -= 1 if count < no_concurrent and not permit_download.is_set(): permit_download.set() async def main(loop): global count permit_download = asyncio.Event() permit_download.set() downloading_event = asyncio.Event() no_concurrent = 3 i = 0 while i < 9: if permit_download.is_set(): count += 1 if count >= no_concurrent: permit_download.clear() loop.create_task(download(i, permit_download, no_concurrent, downloading_event)) await downloading_event.wait() # To force context to switch to download function downloading_event.clear() i += 1 else: await permit_download.wait() await asyncio.sleep(9) if __name__ == '__main__': loop = asyncio.get_event_loop() try: loop.run_until_complete(main(loop)) finally: loop.close() And the output is as expected: downloading 0 will take 2 second(s) downloading 1 will take 3 second(s) downloading 2 will take 1 second(s) downloaded 2 downloading 3 will take 2 second(s) downloaded 0 downloading 4 will take 3 second(s) downloaded 1 downloaded 3 downloading 5 will take 2 second(s) downloading 6 will take 2 second(s) downloaded 5 downloaded 6 downloaded 4 downloading 7 will take 1 second(s) downloading 8 will take 1 second(s) downloaded 7 downloaded 8 But here are my questions: At the moment, I'm simply waiting for 9 seconds to keep the main function running till the downloads are complete. Is there an efficient way of waiting for the last download to complete before exiting the main function? (I know there's asyncio.wait, but I'll need to store all the task references for it to work) What's a good library that does this kind of task? I know javascript has a lot of async libraries, but what about Python? Edit: 2. What's a good library that takes care of common async patterns? (Something like async)

Output: Before reading the rest of this answer, please note that the idiomatic way of limiting the number of parallel tasks this with asyncio is using asyncio.Semaphore, as shown in Mikhail's answer and elegantly encapsulated in Andrei's answer. This answer contains working, but a bit more complicated ways of achieving the same. I am leaving the answer because in some cases this approach can have advantages over a semaphore, specifically when the amount of items to process is very large or unbounded, and you cannot create all the coroutines in advance. In that case the second (queue-based) solution in this answer is what you want. But in most everyday situations, such as parallel download through aiohttp, one should use a semaphore instead. You basically need a fixed-size pool of download tasks. asyncio doesn't come with a pre-made task pool, but it is easy to create one: simply keep a set of tasks and don't allow it to grow past the limit. Although the question states your reluctance to go down that route, the code ends up much more elegant: import asyncio, random async def download(code): wait_time = random.randint(1, 3) print('downloading {} will take {} second(s)'.format(code, wait_time)) await asyncio.sleep(wait_time) # I/O, context will switch to main function print('downloaded {}'.format(code)) async def main(loop): no_concurrent = 3 dltasks = set() i = 0 while i < 9: if len(dltasks) >= no_concurrent: # Wait for some download to finish before adding a new one _done, dltasks = await asyncio.wait( dltasks, return_when=asyncio.FIRST_COMPLETED) dltasks.add(loop.create_task(download(i))) i += 1 # Wait for the remaining downloads to finish await asyncio.wait(dltasks) An alternative is to create a fixed number of coroutines doing the downloading, much like a fixed-size thread pool, and feed them work using an asyncio.Queue. This removes the need to manually limit the number of downloads, which will be automatically limited by the number of coroutines invoking download(): # download() defined as above async def download_worker(q): while True: code = await q.get() await download(code) q.task_done() async def main(loop): q = asyncio.Queue() workers = [loop.create_task(download_worker(q)) for _ in range(3)] i = 0 while i < 9: await q.put(i) i += 1 await q.join() # wait for all tasks to be processed for worker in workers: worker.cancel() await asyncio.gather(*workers, return_exceptions=True) As for your other question, the obvious choice would be aiohttp.


Instruction:
How can I "log in" to a website using Python's Requests module?

I am trying to post a request to log in to a website using the Requests module in Python, but it’s not really working. I'm new to this...so I can't figure out if I should make my Username and Password cookies or some type of HTTP authorization thing I found (??). from pyquery import PyQuery import requests url = 'http://www.locationary.com/home/index2.jsp' So now, I think I'm supposed to use post() and cookies.... ck = {'inUserName': 'USERNAME/EMAIL', 'inUserPass': 'PASSWORD'} r = requests.post(url, cookies=ck) content = r.text q = PyQuery(content) title = q("title").text() print title I have a feeling that I'm doing the cookies thing wrong...I don't know. If it doesn't log in correctly, the title of the home page should come out to "Locationary.com", and if it does, it should be "Home Page." What is the explanation for a few things about Requests and cookies? ...It still didn't really work yet. Okay...so this is what the home page HTML says before you log in: 	 	 <input class="Data_Entry_Field_Login" type="text" name="inUserName" id="inUserName" size="25">	  	 <input class="Data_Entry_Field_Login" type="password" name="inUserPass" id="inUserPass">	 So I think I'm doing it right, but the output is still "Locationary.com" I want to be able to stay logged in for a long time and whenever I request a page under that domain, I want the content to show up as if I were logged in.

Output: If the information you want is on the page you are directed to immediately after login... Let’s call your ck variable payload instead, like in the python-requests documentation: payload = {'inUserName': 'USERNAME/EMAIL', 'inUserPass': 'PASSWORD'} url = 'http://www.locationary.com/home/index2.jsp' requests.post(url, data=payload) Otherwise... See tigerFinch's answer.


Instruction:
Solve Cross Origin Resource Sharing with Flask

For the following ajax post request for Flask (how can I use data posted from ajax in flask?): $.ajax({ url: "http://127.0.0.1:5000/foo", type: "POST", contentType: "application/json", data: JSON.stringify({'inputVar': 1}), success: function( data ) { alert( "success" + data ); } }); I get a Cross Origin Resource Sharing (CORS) error: No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'null' is therefore not allowed access. The response had HTTP status code 500. I tried solving it in the two following ways, but none seems to work. Using Flask-CORS This is a Flask extension for handling CORS that should make cross-origin AJAX possible. http://flask-cors.readthedocs.org/en/latest/ How to enable CORS in flask and heroku Flask-cors wrapper not working when jwt auth wrapper is applied. Javascript - No 'Access-Control-Allow-Origin' header is present on the requested resource My pythonServer.py using this solution: from flask import Flask from flask.ext.cors import CORS, cross_origin app = Flask(__name__) cors = CORS(app, resources={r"/foo": {"origins": "*"}}) app.config['CORS_HEADERS'] = 'Content-Type' @app.route('/foo', methods=['POST','OPTIONS']) @cross_origin(origin='*',headers=['Content-Type','Authorization']) def foo(): return request.json['inputVar'] if __name__ == '__main__': app.run() Using specific Flask Decorator This is an official Flask code snippet defining a decorator that should allow CORS on the functions it decorates. http://flask.pocoo.org/snippets/56/ Python Flask cross site HTTP POST - doesn't work for specific allowed origins http://chopapp.com/#351l7gc3 My pythonServer.py using this solution: from flask import Flask, make_response, request, current_app from datetime import timedelta from functools import update_wrapper app = Flask(__name__) def crossdomain(origin=None, methods=None, headers=None, max_age=21600, attach_to_all=True, automatic_options=True): if methods is not None: methods = ', '.join(sorted(x.upper() for x in methods)) if headers is not None and not isinstance(headers, basestring): headers = ', '.join(x.upper() for x in headers) if not isinstance(origin, basestring): origin = ', '.join(origin) if isinstance(max_age, timedelta): max_age = max_age.total_seconds() def get_methods(): if methods is not None: return methods options_resp = current_app.make_default_options_response() return options_resp.headers['allow'] def decorator(f): def wrapped_function(*args, **kwargs): if automatic_options and request.method == 'OPTIONS': resp = current_app.make_default_options_response() else: resp = make_response(f(*args, **kwargs)) if not attach_to_all and request.method != 'OPTIONS': return resp h = resp.headers h['Access-Control-Allow-Origin'] = origin h['Access-Control-Allow-Methods'] = get_methods() h['Access-Control-Max-Age'] = str(max_age) if headers is not None: h['Access-Control-Allow-Headers'] = headers return resp f.provide_automatic_options = False return update_wrapper(wrapped_function, f) return decorator @app.route('/foo', methods=['GET','POST','OPTIONS']) @crossdomain(origin="*") def foo(): return request.json['inputVar'] if __name__ == '__main__': app.run() Can you please give some some indication of why that is?

Output: It worked like a champ, after bit modification to your code # initialization app = Flask(__name__) app.config['SECRET_KEY'] = 'the quick brown fox jumps over the lazy dog' app.config['CORS_HEADERS'] = 'Content-Type' cors = CORS(app, resources={r"/foo": {"origins": "http://localhost:port"}}) @app.route('/foo', methods=['POST']) @cross_origin(origin='localhost',headers=['Content- Type','Authorization']) def foo(): return request.json['inputVar'] if __name__ == '__main__': app.run() I replaced * by localhost. Since as I read in many blogs and posts, you should allow access for specific domain


Instruction:
Writing a dictionary to a text file?

I have a dictionary and am trying to write it to a file. exDict = {1:1, 2:2, 3:3} with open('file.txt', 'r') as file: file.write(exDict) I then have the error file.write(exDict) TypeError: must be str, not dict So I fixed that error but another error came exDict = {111:111, 222:222} with open('file.txt', 'r') as file: file.write(str(exDict)) The error: file.write(str(exDict)) io.UnsupportedOperation: not writable How do I resolve this issue?

Output: First of all you are opening file in read mode and trying to write into it. Consult - IO modes python Secondly, you can only write a string or bytes to a file. If you want to write a dictionary object, you either need to convert it into string or serialize it. import json # as requested in comment exDict = {'exDict': exDict} with open('file.txt', 'w') as file: file.write(json.dumps(exDict)) # use `json.loads` to do the reverse In case of serialization import cPickle as pickle with open('file.txt', 'w') as file: file.write(pickle.dumps(exDict)) # use `pickle.loads` to do the reverse For python 3.x pickle package import would be different import _pickle as pickle


Instruction:
How to plot multiple bars grouped

How to plot multiple bars in matplotlib, when I tried to call the bar function multiple times, they overlap and as seen the below figure the highest value red can be seen only. How can I plot the multiple bars with dates on the x-axes? So far, I tried this: import matplotlib.pyplot as plt import datetime x = [ datetime.datetime(2011, 1, 4, 0, 0), datetime.datetime(2011, 1, 5, 0, 0), datetime.datetime(2011, 1, 6, 0, 0) ] y = [4, 9, 2] z = [1, 2, 3] k = [11, 12, 13] ax = plt.subplot(111) ax.bar(x, y, width=0.5, color='b', align='center') ax.bar(x, z, width=0.5, color='g', align='center') ax.bar(x, k, width=0.5, color='r', align='center') ax.xaxis_date() plt.show() I got this: The results should be something like, but with the dates are on the x-axes and bars are next to each other:

Output: import matplotlib.pyplot as plt from matplotlib.dates import date2num import datetime x = [ datetime.datetime(2011, 1, 4, 0, 0), datetime.datetime(2011, 1, 5, 0, 0), datetime.datetime(2011, 1, 6, 0, 0) ] x = date2num(x) y = [4, 9, 2] z = [1, 2, 3] k = [11, 12, 13] ax = plt.subplot(111) ax.bar(x-0.2, y, width=0.2, color='b', align='center') ax.bar(x, z, width=0.2, color='g', align='center') ax.bar(x+0.2, k, width=0.2, color='r', align='center') ax.xaxis_date() plt.show() I don't know what's the "y values are also overlapping" means, does the following code solve your problem? ax = plt.subplot(111) w = 0.3 ax.bar(x-w, y, width=w, color='b', align='center') ax.bar(x, z, width=w, color='g', align='center') ax.bar(x+w, k, width=w, color='r', align='center') ax.xaxis_date() ax.autoscale(tight=True) plt.show()


Instruction:
How do you unit test a Celery task?

The Celery documentation mentions testing Celery within Django but doesn't explain how to test a Celery task if you are not using Django. How do you do this?

Output: It is possible to test tasks synchronously using any unittest lib out there. I normally do 2 different test sessions when working with celery tasks. The first one (as I'm suggesting below) is completely synchronous and should be the one that makes sure the algorithm does what it should do. The second session uses the whole system (including the broker) and makes sure I'm not having serialization issues or any other distribution, communication problem. So: from celery import Celery celery = Celery() @celery.task def add(x, y): return x + y And your test: from nose.tools import eq_ def test_add_task(): rst = add.apply(args=(4, 4)).get() eq_(rst, 8)