Instruction:
How to filter Pandas dataframe using 'in' and 'not in' like in SQL

How can I achieve the equivalents of SQL's IN and NOT IN? I have a list with the required values. Here's the scenario: df = pd.DataFrame({'country': ['US', 'UK', 'Germany', 'China']}) countries_to_keep = ['UK', 'China'] # pseudo-code: df[df['country'] not in countries_to_keep] My current way of doing this is as follows: df = pd.DataFrame({'country': ['US', 'UK', 'Germany', 'China']}) df2 = pd.DataFrame({'country': ['UK', 'China'], 'matched': True}) # IN df.merge(df2, how='inner', on='country') # NOT IN not_in = df.merge(df2, how='left', on='country') not_in = not_in[pd.isnull(not_in['matched'])] But this seems like a horrible kludge. Can anyone improve on it?

Output: You can use pd.Series.isin. For "IN" use: something.isin(somewhere) Or for "NOT IN": ~something.isin(somewhere) As a worked example: >>> df country 0 US 1 UK 2 Germany 3 China >>> countries_to_keep ['UK', 'China'] >>> df.country.isin(countries_to_keep) 0 False 1 True 2 False 3 True Name: country, dtype: bool >>> df[df.country.isin(countries_to_keep)] country 1 UK 3 China >>> df[~df.country.isin(countries_to_keep)] country 0 US 2 Germany


Instruction:
What does the Ellipsis object do?

While idly surfing the namespace I noticed an odd looking object called Ellipsis, it does not seem to be or do anything special, but it's a globally available builtin. After a search I found that it is used in some obscure variant of the slicing syntax by Numpy and Scipy... but almost nothing else. Was this object added to the language specifically to support Numpy + Scipy? Does Ellipsis have any generic meaning or use at all? D:\workspace\numpy>python Python 2.4.4 (#71, Oct 18 2006, 08:34:43) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> Ellipsis Ellipsis

Output: This came up in another question recently. I'll elaborate on my answer from there: Ellipsis is an object that can appear in slice notation. For example: myList[1:2, ..., 0] Its interpretation is purely up to whatever implements the __getitem__ function and sees Ellipsis objects there, but its main (and intended) use is in the numpy third-party library, which adds a multidimensional array type. Since there are more than one dimensions, slicing becomes more complex than just a start and stop index; it is useful to be able to slice in multiple dimensions as well. E.g., given a 4 × 4 array, the top left area would be defined by the slice [:2, :2]: >>> a array([[ 1, 2, 3, 4], [ 5, 6, 7, 8], [ 9, 10, 11, 12], [13, 14, 15, 16]]) >>> a[:2, :2] # top left array([[1, 2], [5, 6]]) Extending this further, Ellipsis is used here to indicate a placeholder for the rest of the array dimensions not specified. Think of it as indicating the full slice [:] for all the dimensions in the gap it is placed, so for a 3d array, a[..., 0] is the same as a[:, :, 0] and for 4d a[:, :, :, 0], similarly, a[0, ..., 0] is a[0, :, :, 0] (with however many colons in the middle make up the full number of dimensions in the array). Interestingly, in python3, the Ellipsis literal (...) is usable outside the slice syntax, so you can actually write: >>> ... Ellipsis EDIT: Ellipsis is also used in the standard library typing module: e.g. Callable[..., int] to indicate a callable that returns an int without specifying the signature, or Tuple[str, ...] to indicate a variable-length homogeneous tuple of strings.


Instruction:
How do I generate all permutations of a list?

How do I generate all the permutations of a list? For example: permutations([]) [] permutations([1]) [1] permutations([1, 2]) [1, 2] [2, 1] permutations([1, 2, 3]) [1, 2, 3] [1, 3, 2] [2, 1, 3] [2, 3, 1] [3, 1, 2] [3, 2, 1]

Output: Use itertools.permutations from the standard library: import itertools list(itertools.permutations([1, 2, 3])) A demonstration of how itertools.permutations might be implemented: def permutations(elements): if len(elements) <= 1: yield elements return for perm in permutations(elements[1:]): for i in range(len(elements)): # nb elements[0:1] works in both string and list contexts yield perm[:i] + elements[0:1] + perm[i:] A couple of alternative approaches are listed in the documentation of itertools.permutations. Here's one: def permutations(iterable, r=None): # permutations('ABCD', 2) --> AB AC AD BA BC BD CA CB CD DA DB DC # permutations(range(3)) --> 012 021 102 120 201 210 pool = tuple(iterable) n = len(pool) r = n if r is None else r if r > n: return indices = range(n) cycles = range(n, n-r, -1) yield tuple(pool[i] for i in indices[:r]) while n: for i in reversed(range(r)): cycles[i] -= 1 if cycles[i] == 0: indices[i:] = indices[i+1:] + indices[i:i+1] cycles[i] = n - i else: j = cycles[i] indices[i], indices[-j] = indices[-j], indices[i] yield tuple(pool[i] for i in indices[:r]) break else: return And another, based on itertools.product: def permutations(iterable, r=None): pool = tuple(iterable) n = len(pool) r = n if r is None else r for indices in product(range(n), repeat=r): if len(set(indices)) == r: yield tuple(pool[i] for i in indices)


Instruction:
How to update/upgrade a package using pip?

What is the way to update a package using pip? These commands do not work: pip update pip upgrade I know this is a simple question but it is needed as it is not so easy to find (pip documentation doesn't pop up and other questions from stack overflow are relevant but are not exactly about that)

Output: This is the way pip install <package_name> --upgrade or in short pip install <package_name> -U Using sudo will ask to enter your root password to confirm the action, but although common, is considered unsafe. If you do not have a root password (if you are not the admin) you should probably work with virtualenv. You can also use the user flag to install it on this user only. pip install <package_name> --upgrade --user


Instruction:
Constructing DataFrame from values in variables yields "ValueError: If using all scalar values, you must pass an index"

I have two variables as follows. a = 2 b = 3 I want to construct a DataFrame from this: df2 = pd.DataFrame({'A':a, 'B':b}) This generates an error: ValueError: If using all scalar values, you must pass an index I tried this also: df2 = (pd.DataFrame({'a':a, 'b':b})).reset_index() This gives the same error message. How do I do what I want?

Output: The error message says that if you're passing scalar values, you have to pass an index. So you can either not use scalar values for the columns -- e.g. use a list: >>> df = pd.DataFrame({'A': [a], 'B': [b]}) >>> df A B 0 2 3 or use scalar values and pass an index: >>> df = pd.DataFrame({'A': a, 'B': b}, index=[0, 3]) >>> df A B 0 2 3 3 2 3


Instruction:
How do I change the working directory in Python?

cd is the shell command to change the working directory. What is the equivalent in Python?

Output: You can change the working directory with: import os os.chdir(path) You should be careful that changing the directory may result in destructive changes your code applies in the new location. Potentially worse still, do not catch exceptions such as WindowsError and OSError after changing directory as that may mean destructive changes are applied in the old location! If you're on Python 3.11 or newer, then consider using this context manager to ensure you return to the original working directory when you're done: from contextlib import chdir with chdir(path): # do stuff here If you're on an older version of Python, Brian M. Hunt's answer shows how to roll your own context manager: his answer. Changing the current working directory in a subprocess does not change the current working directory in the parent process. This is true of the Python interpreter as well. You cannot use os.chdir() to change the CWD of the calling process.


Instruction:
How to convert index of a pandas dataframe into a column

How to convert an index of a dataframe into a column? For example: gi ptt_loc 0 384444683 593 1 384444684 594 2 384444686 596 to index1 gi ptt_loc 0 0 384444683 593 1 1 384444684 594 2 2 384444686 596

Output: either: df['index1'] = df.index or .reset_index: df = df.reset_index() If you have a multi-index frame with 3 levels of index, like: >>> df val tick tag obs 2016-02-26 C 2 0.0139 2016-02-27 A 2 0.5577 2016-02-28 C 6 0.0303 and you want to convert the 1st (tick) and 3rd (obs) levels in the index into columns, you could do: >>> df.reset_index(level=['tick', 'obs']) tick obs val tag C 2016-02-26 2 0.0139 A 2016-02-27 2 0.5577 C 2016-02-28 6 0.0303


Instruction:
How to state in requirements.txt a direct github source

I've installed a library using the command pip install git+git://github.com/mozilla/elasticutils.git which installs it directly from a Github repository. This works fine and I want to have that dependency in my requirements.txt. I've looked at other tickets like this but that didn't solve my problem. If I put something like -f git+git://github.com/mozilla/elasticutils.git elasticutils==0.7.dev in the requirements.txt file, a pip install -r requirements.txt results in the following output: Downloading/unpacking elasticutils==0.7.dev (from -r requirements.txt (line 20)) Could not find a version that satisfies the requirement elasticutils==0.7.dev (from -r requirements.txt (line 20)) (from versions: ) No distributions matching the version for elasticutils==0.7.dev (from -r requirements.txt (line 20)) The documentation of the requirements file does not mention links using the git+git protocol specifier, so maybe this is just not supported. Does anybody have a solution for my problem?

Output: Normally your requirements.txt file would look something like this: package-one==1.9.4 package-two==3.7.1 package-three==1.0.1 ... To specify a Github repo, you do not need the package-name== convention. The examples below update package-two using a GitHub repo. The text after @ denotes the specifics of the package. Specify commit hash (41b95ec in the context of updated requirements.txt): package-one==1.9.4 package-two @ git+https://github.com/owner/repo@41b95ec package-three==1.0.1 Specify branch name (main): package-two @ git+https://github.com/owner/repo@main Specify tag (0.1): package-two @ git+https://github.com/owner/repo@0.1 Specify release (3.7.1): package-two @ git+https://github.com/owner/repo@releases/tag/v3.7.1 Note that in certain versions of pip you will need to update the package version in the package's setup.py, or pip will assume the requirement is already satisfied and not install the new version. For instance, if you have 1.2.1 installed, and want to fork this package with your own version, you could use the above technique in your requirements.txt and then update setup.py to 1.2.1.1. See also the pip documentation on VCS support.


Instruction:
How to serve static files in Flask

I've got an application that I threw together in Flask and for now it is just serving up a single static HTML page with some links to CSS and JS. And I can't find where in the documentation Flask describes returning static files. Yes, I could use render_template but I know the data is not templatized. I'd have thought send_file or url_for was the right thing, but I could not get those to work. In the meantime, I am opening the files, reading content, and rigging up a Response with appropriate mimetype: import os.path from flask import Flask, Response app = Flask(__name__) app.config.from_object(__name__) def root_dir(): # pragma: no cover return os.path.abspath(os.path.dirname(__file__)) def get_file(filename): # pragma: no cover try: src = os.path.join(root_dir(), filename) # Figure out how flask returns static files # Tried: # - render_template # - send_file # This should not be so non-obvious return open(src).read() except IOError as exc: return str(exc) @app.route('/', methods=['GET']) def metrics(): # pragma: no cover content = get_file('jenkins_analytics.html') return Response(content, mimetype="text/html") @app.route('/', defaults={'path': ''}) @app.route('/<path:path>') def get_resource(path): # pragma: no cover mimetypes = { ".css": "text/css", ".html": "text/html", ".js": "application/javascript", } complete_path = os.path.join(root_dir(), path) ext = os.path.splitext(path)[1] mimetype = mimetypes.get(ext, "text/html") content = get_file(complete_path) return Response(content, mimetype=mimetype) if __name__ == '__main__': # pragma: no cover app.run(port=80) Someone want to give a code sample or url for this?

Output: In production, configure the HTTP server (Nginx, Apache, etc.) in front of your application to serve requests to /static from the static folder. A dedicated web server is very good at serving static files efficiently, although you probably won't notice a difference compared to Flask at low volumes. Flask automatically creates a /static/<path:filename> route that will serve any filename under the static folder next to the Python module that defines your Flask app. Use url_for to link to static files: url_for('static', filename='js/analytics.js') You can also use send_from_directory to serve files from a directory in your own route. This takes a base directory and a path, and ensures that the path is contained in the directory, which makes it safe to accept user-provided paths. This can be useful in cases where you want to check something before serving the file, such as if the logged in user has permission. from flask import send_from_directory @app.route('/reports/<path:path>') def send_report(path): # Using request args for path will expose you to directory traversal attacks return send_from_directory('reports', path) WARNING: Do not use send_file or send_static_file with a user-supplied path. This will expose you to directory traversal attacks. send_from_directory was designed to safely handle user-supplied paths under a known directory, and will raise an error if the path attempts to escape the directory. If you are generating a file in memory without writing it to the filesystem, you can pass a BytesIO object to send_file to serve it like a file. You'll need to pass other arguments to send_file in this case since it can't infer things like the file name or content type.


Instruction:
How to combine multiple QuerySets in Django?

I'm trying to build the search for a Django site I am building, and in that search, I am searching across three different models. And to get pagination on the search result list, I would like to use a generic object_list view to display the results. But to do that, I have to merge three QuerySets into one. How can I do that? I've tried this: result_list = [] page_list = Page.objects.filter( Q(title__icontains=cleaned_search_term) | Q(body__icontains=cleaned_search_term)) article_list = Article.objects.filter( Q(title__icontains=cleaned_search_term) | Q(body__icontains=cleaned_search_term) | Q(tags__icontains=cleaned_search_term)) post_list = Post.objects.filter( Q(title__icontains=cleaned_search_term) | Q(body__icontains=cleaned_search_term) | Q(tags__icontains=cleaned_search_term)) for x in page_list: result_list.append(x) for x in article_list: result_list.append(x) for x in post_list: result_list.append(x) return object_list( request, queryset=result_list, template_object_name='result', paginate_by=10, extra_context={ 'search_term': search_term}, template_name="search/result_list.html") But this doesn't work. I get an error when I try to use that list in the generic view. The list is missing the clone attribute. How can I merge the three lists, page_list, article_list and post_list?

Output: Concatenating the querysets into a list is the simplest approach. If the database will be hit for all querysets anyway (e.g. because the result needs to be sorted), this won't add further cost. from itertools import chain result_list = list(chain(page_list, article_list, post_list)) Using itertools.chain is faster than looping each list and appending elements one by one, since itertools is implemented in C. It also consumes less memory than converting each queryset into a list before concatenating. Now it's possible to sort the resulting list e.g. by date (as requested in hasen j's comment to another answer). The sorted() function conveniently accepts a generator and returns a list: from operator import attrgetter result_list = sorted( chain(page_list, article_list, post_list), key=attrgetter('date_created') ) You can reverse the sort order: result_list = sorted( chain(page_list, article_list, post_list), key=attrgetter('date_created'), reverse=True, ) attrgetter is equivalet to the following lambda (this was the way it had to be done before Python 2.4): result_list = sorted( chain(page_list, article_list, post_list), key=lambda instance: instance.date_created, )


Instruction:
How to urlencode a querystring in Python?

I am trying to urlencode this string before I submit. queryString = 'eventName=' + evt.fields["eventName"] + '&' + 'eventDescription=' + evt.fields["eventDescription"];

Output: Python 3 Use urllib.parse.urlencode: >>> import urllib.parse >>> f = { 'eventName' : 'myEvent', 'eventDescription' : 'cool event'} >>> urllib.parse.urlencode(f) eventName=myEvent&eventDescription=cool+event Note that this does not do url encoding in the commonly used sense (look at the output). For that use urllib.parse.quote_plus. Python 2 You need to pass your parameters into urllib.urlencode() as either a mapping (dict), or a sequence of 2-tuples, like: >>> import urllib >>> f = { 'eventName' : 'myEvent', 'eventDescription' : 'cool event'} >>> urllib.urlencode(f) 'eventName=myEvent&eventDescription=cool+event'


Instruction:
How can I pass a list as a command-line argument with argparse?

I am trying to pass a list as an argument to a command line program. Is there an argparse option to pass a list as option? parser.add_argument('-l', '--list', type=list, action='store', dest='list', help='<Required> Set flag', required=True) Script is called like below python test.py -l "265340 268738 270774 270817"

Output: SHORT ANSWER Use the nargs option or the 'append' setting of the action option (depending on how you want the user interface to behave). nargs parser.add_argument('-l','--list', nargs='+', help='<Required> Set flag', required=True) # Use like: # python arg.py -l 1234 2345 3456 4567 nargs='+' takes 1 or more arguments, nargs='*' takes zero or more. append parser.add_argument('-l','--list', action='append', help='<Required> Set flag', required=True) # Use like: # python arg.py -l 1234 -l 2345 -l 3456 -l 4567 With append you provide the option multiple times to build up the list. Don't use type=list!!! - There is probably no situation where you would want to use type=list with argparse. Ever. LONG ANSWER Let's take a look in more detail at some of the different ways one might try to do this, and the end result. import argparse parser = argparse.ArgumentParser() # By default it will fail with multiple arguments. parser.add_argument('--default') # Telling the type to be a list will also fail for multiple arguments, # but give incorrect results for a single argument. parser.add_argument('--list-type', type=list) # This will allow you to provide multiple arguments, but you will get # a list of lists which is not desired. parser.add_argument('--list-type-nargs', type=list, nargs='+') # This is the correct way to handle accepting multiple arguments. # '+' == 1 or more. # '*' == 0 or more. # '?' == 0 or 1. # An int is an explicit number of arguments to accept. parser.add_argument('--nargs', nargs='+') # To make the input integers parser.add_argument('--nargs-int-type', nargs='+', type=int) # An alternate way to accept multiple inputs, but you must # provide the flag once per input. Of course, you can use # type=int here if you want. parser.add_argument('--append-action', action='append') # To show the results of the given option to screen. for _, value in parser.parse_args()._get_kwargs(): if value is not None: print(value) Here is the output you can expect: $ python arg.py --default 1234 2345 3456 4567 ... arg.py: error: unrecognized arguments: 2345 3456 4567 $ python arg.py --list-type 1234 2345 3456 4567 ... arg.py: error: unrecognized arguments: 2345 3456 4567 $ # Quotes won't help here... $ python arg.py --list-type "1234 2345 3456 4567" ['1', '2', '3', '4', ' ', '2', '3', '4', '5', ' ', '3', '4', '5', '6', ' ', '4', '5', '6', '7'] $ python arg.py --list-type-nargs 1234 2345 3456 4567 [['1', '2', '3', '4'], ['2', '3', '4', '5'], ['3', '4', '5', '6'], ['4', '5', '6', '7']] $ python arg.py --nargs 1234 2345 3456 4567 ['1234', '2345', '3456', '4567'] $ python arg.py --nargs-int-type 1234 2345 3456 4567 [1234, 2345, 3456, 4567] $ # Negative numbers are handled perfectly fine out of the box. $ python arg.py --nargs-int-type -1234 2345 -3456 4567 [-1234, 2345, -3456, 4567] $ python arg.py --append-action 1234 --append-action 2345 --append-action 3456 --append-action 4567 ['1234', '2345', '3456', '4567'] Takeaways: Use nargs or action='append' nargs can be more straightforward from a user perspective, but it can be unintuitive if there are positional arguments because argparse can't tell what should be a positional argument and what belongs to the nargs; if you have positional arguments then action='append' may end up being a better choice. The above is only true if nargs is given '*', '+', or '?'. If you provide an integer number (such as 4) then there will be no problem mixing options with nargs and positional arguments because argparse will know exactly how many values to expect for the option. Don't use quotes on the command line1 Don't use type=list, as it will return a list of lists This happens because under the hood argparse uses the value of type to coerce each individual given argument you your chosen type, not the aggregate of all arguments. You can use type=int (or whatever) to get a list of ints (or whatever) 1: I don't mean in general.. I mean using quotes to pass a list to argparse is not what you want.


Instruction:
How to print a date in a regular format?

This is my code: import datetime today = datetime.date.today() print(today) This prints: 2008-11-22 which is exactly what I want. But, I have a list I'm appending this to and then suddenly everything goes "wonky". Here is the code: import datetime mylist = [datetime.date.today()] print(mylist) This prints [datetime.date(2008, 11, 22)]. How can I get just a simple date like 2008-11-22?

Output: The WHY: dates are objects In Python, dates are objects. Therefore, when you manipulate them, you manipulate objects, not strings or timestamps. Any object in Python has TWO string representations: The regular representation that is used by print can be get using the str() function. It is most of the time the most common human readable format and is used to ease display. So str(datetime.datetime(2008, 11, 22, 19, 53, 42)) gives you '2008-11-22 19:53:42'. The alternative representation that is used to represent the object nature (as a data). It can be get using the repr() function and is handy to know what kind of data your manipulating while you are developing or debugging. repr(datetime.datetime(2008, 11, 22, 19, 53, 42)) gives you 'datetime.datetime(2008, 11, 22, 19, 53, 42)'. What happened is that when you have printed the date using print, it used str() so you could see a nice date string. But when you have printed mylist, you have printed a list of objects and Python tried to represent the set of data, using repr(). The How: what do you want to do with that? Well, when you manipulate dates, keep using the date objects all long the way. They got thousand of useful methods and most of the Python API expect dates to be objects. When you want to display them, just use str(). In Python, the good practice is to explicitly cast everything. So just when it's time to print, get a string representation of your date using str(date). One last thing. When you tried to print the dates, you printed mylist. If you want to print a date, you must print the date objects, not their container (the list). E.G, you want to print all the date in a list : for date in mylist : print str(date) Note that in that specific case, you can even omit str() because print will use it for you. But it should not become a habit :-) Practical case, using your code import datetime mylist = [] today = datetime.date.today() mylist.append(today) print mylist[0] # print the date object, not the container ;-) 2008-11-22 # It's better to always use str() because : print "This is a new day : ", mylist[0] # will work >>> This is a new day : 2008-11-22 print "This is a new day : " + mylist[0] # will crash >>> cannot concatenate 'str' and 'datetime.date' objects print "This is a new day : " + str(mylist[0]) >>> This is a new day : 2008-11-22 Advanced date formatting Dates have a default representation, but you may want to print them in a specific format. In that case, you can get a custom string representation using the strftime() method. strftime() expects a string pattern explaining how you want to format your date. E.G : print today.strftime('We are the %d, %b %Y') >>> 'We are the 22, Nov 2008' All the letter after a "%" represent a format for something: %d is the day number (2 digits, prefixed with leading zero's if necessary) %m is the month number (2 digits, prefixed with leading zero's if necessary) %b is the month abbreviation (3 letters) %B is the month name in full (letters) %y is the year number abbreviated (last 2 digits) %Y is the year number full (4 digits) etc. Have a look at the official documentation, or McCutchen's quick reference you can't know them all. Since PEP3101, every object can have its own format used automatically by the method format of any string. In the case of the datetime, the format is the same used in strftime. So you can do the same as above like this: print "We are the {:%d, %b %Y}".format(today) >>> 'We are the 22, Nov 2008' The advantage of this form is that you can also convert other objects at the same time. With the introduction of Formatted string literals (since Python 3.6, 2016-12-23) this can be written as import datetime f"{datetime.datetime.now():%Y-%m-%d}" >>> '2017-06-15' Localization Dates can automatically adapt to the local language and culture if you use them the right way, but it's a bit complicated. Maybe for another question on SO(Stack Overflow) ;-)


Instruction:
Split string with multiple delimiters in Python

I found some answers online, but I have no experience with regular expressions, which I believe is what is needed here. I have a string that needs to be split by either a ';' or ', ' That is, it has to be either a semicolon or a comma followed by a space. Individual commas without trailing spaces should be left untouched Example string: "b-staged divinylsiloxane-bis-benzocyclobutene [124221-30-3], mesitylene [000108-67-8]; polymerized 1,2-dihydro-2,2,4- trimethyl quinoline [026780-96-1]" should be split into a list containing the following: ('b-staged divinylsiloxane-bis-benzocyclobutene [124221-30-3]' , 'mesitylene [000108-67-8]', 'polymerized 1,2-dihydro-2,2,4- trimethyl quinoline [026780-96-1]')

Output: Luckily, Python has this built-in :) import re # Regex pattern splits on substrings "; " and ", " re.split('; |, ', string_to_split) Update: Following your comment: >>> string_to_split = 'Beautiful, is; better*than\nugly' >>> import re >>> re.split('; |, |\*|\n', string_to_split) ['Beautiful', 'is', 'better', 'than', 'ugly']


Instruction:
Filter dict to contain only certain keys?

I've got a dict that has a whole bunch of entries. I'm only interested in a select few of them. Is there an easy way to prune all the other ones out?

Output: Constructing a new dict: dict_you_want = {key: old_dict[key] for key in your_keys} Uses dictionary comprehension. If you use a version which lacks them (ie Python 2.6 and earlier), make it dict((key, old_dict[key]) for ...). It's the same, though uglier. Note that this, unlike jnnnnn's version, has stable performance (depends only on number of your_keys) for old_dicts of any size. Both in terms of speed and memory. Since this is a generator expression, it processes one item at a time, and it doesn't looks through all items of old_dict. Removing everything in-place: unwanted = set(old_dict) - set(your_keys) for unwanted_key in unwanted: del your_dict[unwanted_key]


Instruction:
Why shouldn't I use PyPy over CPython if PyPy is 6.3 times faster?

I've been hearing a lot about the PyPy project. They claim it is 6.3 times faster than the CPython interpreter on their site. Whenever we talk about dynamic languages like Python, speed is one of the top issues. To solve this, they say PyPy is 6.3 times faster. The second issue is parallelism, the infamous Global Interpreter Lock (GIL). For this, PyPy says it can give GIL-less Python. If PyPy can solve these great challenges, what are its weaknesses that are preventing wider adoption? That is to say, what's preventing someone like me, a typical Python developer, from switching to PyPy right now?

Output: NOTE: PyPy is more mature and better supported now than it was in 2013, when this question was asked. Avoid drawing conclusions from out-of-date information. PyPy, as others have been quick to mention, has tenuous support for C extensions. It has support, but typically at slower-than-Python speeds and it's iffy at best. Hence a lot of modules simply require CPython. Check the list of supported packages, but look at the date that list was updated, because it's not not kept in lockstep with actual support, so it's still possible that packages that marked unsupported on that list are actually supported. Python support typically lags a few versions behind, so if you absolutely need the latest features, you may need to wait a while before PyPy supports them. PyPy sometimes isn't actually faster for "scripts", which a lot of people use Python for. These are the short-running programs that do something simple and small. Because PyPy is a JIT compiler its main advantages come from long run times and simple types (such as numbers). PyPy's pre-JIT speeds can be bad compared to CPython. Inertia. Moving to PyPy often requires retooling, which for some people and organizations is simply too much work. Those are the main reasons that affect me, I'd say.


Instruction:
Convert all strings in a list to integers

How do I convert all strings in a list to integers? ['1', '2', '3'] ⟶ [1, 2, 3]

Output: Given: xs = ['1', '2', '3'] Use map then list to obtain a list of integers: list(map(int, xs)) In Python 2, list was unnecessary since map returned a list: map(int, xs)


Instruction:
What do ** (double star/asterisk) and * (star/asterisk) mean in a function call?

In code like zip(*x) or f(**k), what do the * and ** respectively mean? How does Python implement that behaviour, and what are the performance implications? See also: Expanding tuples into arguments. Please use that one to close questions where OP needs to use * on an argument and doesn't know it exists. Similarly, use Converting Python dict to kwargs? for the case of using **. See What does ** (double star/asterisk) and * (star/asterisk) do for parameters? for the complementary question about parameters. See What do ** (double star/asterisk) and * (star/asterisk) inside square brakets mean for class and function declarations in Python 3.12+? for the complementary question when used for type parameters.

Output: A single star * unpacks a sequence or collection into positional arguments. Suppose we have def add(a, b): return a + b values = (1, 2) Using the * unpacking operator, we can write s = add(*values), which will be equivalent to writing s = add(1, 2). The double star ** does the same thing for a dictionary, providing values for named arguments: values = {'a': 1, 'b': 2} s = add(**values) # equivalent to add(a=1, b=2) Both operators can be used for the same function call. For example, given: def sum(a, b, c, d): return a + b + c + d values1 = (1, 2) values2 = {'c': 10, 'd': 15} then s = add(*values1, **values2) is equivalent to s = sum(1, 2, c=10, d=15). See also the relevant section of the tutorial in the Python documentation. Similarly, * and ** can be used for parameters. Using * allows a function to accept any number of positional arguments, which will be collected into a single parameter: def add(*values): s = 0 for v in values: s = s + v return s Now when the function is called like s = add(1, 2, 3, 4, 5), values will be the tuple (1, 2, 3, 4, 5) (which, of course, produces the result 15). Similarly, a parameter marked with ** will receive a dict: def get_a(**values): return values['a'] s = get_a(a=1, b=2) # returns 1 this allows for specifying a large number of optional parameters without having to declare them. Again, both can be combined: def add(*values, **options): s = 0 for i in values: s = s + i if "neg" in options: if options["neg"]: s = -s return s s = add(1, 2, 3, 4, 5) # returns 15 s = add(1, 2, 3, 4, 5, neg=True) # returns -15 s = add(1, 2, 3, 4, 5, neg=False) # returns 15


Instruction:
IndentationError: unindent does not match any outer indentation level, although the indentation looks correct

When I compile the Python code below, I get IndentationError: unindent does not match any outer indentation level import sys def Factorial(n): # Return factorial result = 1 for i in range (1,n): result = result * i print "factorial is ",result return result Why?

Output: One possible cause for this error is that there might be spaces mixed with tabs for indentation. Try doing a search & replace to replace all tabs with a few spaces. Try this: import sys def Factorial(n): # return factorial result = 1 for i in range (1,n): result = result * i print "factorial is ",result return result print Factorial(10)


Instruction:
Return a default value if a dictionary key is not available

I need a way to get a dictionary value if its key exists, or simply return None, if it does not. However, Python raises a KeyError exception if you search for a key that does not exist. I know that I can check for the key, but I am looking for something more explicit. Is there a way to just return None if the key does not exist? See also: Why dict.get(key) instead of dict[key]?

Output: You can use dict.get() value = d.get(key) which will return None if key is not in d. You can also provide a different default value that will be returned instead of None (unless the dict actually contains None as the value for this key): value = d.get(key, "empty")


Instruction:
Determine function name from within that function

Is there a way to determine a function's name from within the function? def foo(): print("my name is", __myname__) # <== how do I calculate this at runtime? In the example above, the body of foo will somehow access the function name "foo" without hard-coding it. The output would be: >>> foo() my name is foo

Output: If you don't want to play with the stack yourself, you should either use "bar" or bar.__name__ depending on context. Python doesn't have a feature to access the function or its name within the function itself. A magic __function__ had been proposed for Python 3.0 but rejected. See PEP 3130 – Access to Current Module/Class/Function. The given rejection notice is: This PEP is rejected. It is not clear how it should be implemented or what the precise semantics should be in edge cases, and there aren't enough important use cases given. response has been lukewarm at best.


Instruction:
Does Python have “private” variables in classes?

I'm coming from the Java world and reading Bruce Eckels' Python 3 Patterns, Recipes and Idioms. While reading about classes, it goes on to say that in Python there is no need to declare instance variables. You just use them in the constructor, and boom, they are there. So for example: class Simple: def __init__(self, s): print("inside the simple constructor") self.s = s def show(self): print(self.s) def showMsg(self, msg): print(msg + ':', self.show()) If that’s true, then any object of class Simple can just change the value of variable s outside of the class. For example: if __name__ == "__main__": x = Simple("constructor argument") x.s = "test15" # this changes the value x.show() x.showMsg("A message") In Java, we have been taught about public/private/protected variables. Those keywords make sense because at times you want variables in a class to which no one outside the class has access to. Why is that not required in Python?

Output: It's cultural. In Python, you don't write to other classes' instance or class variables. In Java, nothing prevents you from doing the same if you really want to - after all, you can always edit the source of the class itself to achieve the same effect. Python drops that pretence of security and encourages programmers to be responsible. In practice, this works very nicely. If you want to emulate private variables for some reason, you can always use the __ prefix from PEP 8. Python mangles the names of variables like __foo so that they're not easily visible to code outside the namespace that contains them (although you can get around it if you're determined enough, just like you can get around Java's protections if you work at it). By the same convention, the _ prefix means _variable should be used internally in the class (or module) only, even if you're not technically prevented from accessing it from somewhere else. You don't play around with another class's variables that look like __foo or _bar.


Instruction:
Import multiple CSV files into pandas and concatenate into one DataFrame

I would like to read several CSV files from a directory into pandas and concatenate them into one big DataFrame. I have not been able to figure it out though. Here is what I have so far: import glob import pandas as pd # Get data file names path = r'C:\DRO\DCL_rawdata_files' filenames = glob.glob(path + "/*.csv") dfs = [] for filename in filenames: dfs.append(pd.read_csv(filename)) # Concatenate all data into one DataFrame big_frame = pd.concat(dfs, ignore_index=True) I guess I need some help within the for loop?

Output: See pandas: IO tools for all of the available .read_ methods. Try the following code if all of the CSV files have the same columns. I have added header=0, so that after reading the CSV file's first row, it can be assigned as the column names. import pandas as pd import glob import os path = r'C:\DRO\DCL_rawdata_files' # use your path all_files = glob.glob(os.path.join(path , "/*.csv")) li = [] for filename in all_files: df = pd.read_csv(filename, index_col=None, header=0) li.append(df) frame = pd.concat(li, axis=0, ignore_index=True) Or, with attribution to a comment from Sid. all_files = glob.glob(os.path.join(path, "*.csv")) df = pd.concat((pd.read_csv(f) for f in all_files), ignore_index=True) It's often necessary to identify each sample of data, which can be accomplished by adding a new column to the dataframe. pathlib from the standard library will be used for this example. It treats paths as objects with methods, instead of strings to be sliced. Imports and Setup from pathlib import Path import pandas as pd import numpy as np path = r'C:\DRO\DCL_rawdata_files' # or unix / linux / mac path # Get the files from the path provided in the OP files = Path(path).glob('*.csv') # .rglob to get subdirectories Option 1: Add a new column with the file name dfs = list() for f in files: data = pd.read_csv(f) # .stem is method for pathlib objects to get the filename w/o the extension data['file'] = f.stem dfs.append(data) df = pd.concat(dfs, ignore_index=True) Option 2: Add a new column with a generic name using enumerate dfs = list() for i, f in enumerate(files): data = pd.read_csv(f) data['file'] = f'File {i}' dfs.append(data) df = pd.concat(dfs, ignore_index=True) Option 3: Create the dataframes with a list comprehension, and then use np.repeat to add a new column. [f'S{i}' for i in range(len(dfs))] creates a list of strings to name each dataframe. [len(df) for df in dfs] creates a list of lengths Attribution for this option goes to this plotting answer. # Read the files into dataframes dfs = [pd.read_csv(f) for f in files] # Combine the list of dataframes df = pd.concat(dfs, ignore_index=True) # Add a new column df['Source'] = np.repeat([f'S{i}' for i in range(len(dfs))], [len(df) for df in dfs]) Option 4: One liners using .assign to create the new column, with attribution to a comment from C8H10N4O2 df = pd.concat((pd.read_csv(f).assign(filename=f.stem) for f in files), ignore_index=True) or df = pd.concat((pd.read_csv(f).assign(Source=f'S{i}') for i, f in enumerate(files)), ignore_index=True)


Instruction:
When to use cla(), clf() or close() for clearing a plot

Matplotlib offers these functions: cla() # Clear axis clf() # Clear figure close() # Close a figure window When should I use each function and what exactly does it do?

Output: They all do different things, since matplotlib uses a hierarchical order in which a figure window contains a figure which may consist of many axes. Additionally, there are functions from the pyplot interface and there are methods on the Figure class. I will discuss both cases below. pyplot interface pyplot is a module that collects a couple of functions that allow matplotlib to be used in a functional manner. I here assume that pyplot has been imported as import matplotlib.pyplot as plt. In this case, there are three different commands that remove stuff: See matplotlib.pyplot Functions: plt.cla() clears an axis, i.e. the currently active axis in the current figure. It leaves the other axes untouched. plt.clf() clears the entire current figure with all its axes, but leaves the window opened, such that it may be reused for other plots. plt.close() closes a window, which will be the current window, if not specified otherwise. Which functions suits you best depends thus on your use-case. The close() function furthermore allows one to specify which window should be closed. The argument can either be a number or name given to a window when it was created using figure(number_or_name) or it can be a figure instance fig obtained, i.e., usingfig = figure(). If no argument is given to close(), the currently active window will be closed. Furthermore, there is the syntax close('all'), which closes all figures. methods of the Figure class Additionally, the Figure class provides methods for clearing figures. I'll assume in the following that fig is an instance of a Figure: fig.clear() clears the entire figure, and is in the primary list of functions to get and modify the figure state. This call is equivalent to plt.clf() only if fig is the current figure. fig.clf() is a synonym for fig.clear() but is discouraged. Note that even del fig will not close the associated figure window. As far as I know the only way to close a figure window is using plt.close(fig) as described above.


Instruction:
How do I append one string to another in Python?

How do I efficiently append one string to another? Are there any faster alternatives to: var1 = "foo" var2 = "bar" var3 = var1 + var2 For handling multiple strings in a list, see How to concatenate (join) items in a list to a single string. See How do I put a variable’s value inside a string (interpolate it into the string)? if some inputs are not strings, but the result should still be a string.

Output: If you only have one reference to a string and you concatenate another string to the end, CPython now special cases this and tries to extend the string in place. The end result is that the operation is amortized O(n). e.g. s = "" for i in range(n): s += str(i) used to be O(n^2), but now it is O(n). More information From the source (bytesobject.c): void PyBytes_ConcatAndDel(register PyObject **pv, register PyObject *w) { PyBytes_Concat(pv, w); Py_XDECREF(w); } /* The following function breaks the notion that strings are immutable: it changes the size of a string. We get away with this only if there is only one module referencing the object. You can also think of it as creating a new string object and destroying the old one, only more efficiently. In any case, don't use this if the string may already be known to some other part of the code... Note that if there's not enough memory to resize the string, the original string object at *pv is deallocated, *pv is set to NULL, an "out of memory" exception is set, and -1 is returned. Else (on success) 0 is returned, and the value in *pv may or may not be the same as on input. As always, an extra byte is allocated for a trailing \0 byte (newsize does *not* include that), and a trailing \0 byte is stored. */ int _PyBytes_Resize(PyObject **pv, Py_ssize_t newsize) { register PyObject *v; register PyBytesObject *sv; v = *pv; if (!PyBytes_Check(v) || Py_REFCNT(v) != 1 || newsize < 0) { *pv = 0; Py_DECREF(v); PyErr_BadInternalCall(); return -1; } /* XXX UNREF/NEWREF interface should be more symmetrical */ _Py_DEC_REFTOTAL; _Py_ForgetReference(v); *pv = (PyObject *) PyObject_REALLOC((char *)v, PyBytesObject_SIZE + newsize); if (*pv == NULL) { PyObject_Del(v); PyErr_NoMemory(); return -1; } _Py_NewReference(*pv); sv = (PyBytesObject *) *pv; Py_SIZE(sv) = newsize; sv->ob_sval[newsize] = '\0'; sv->ob_shash = -1; /* invalidate cached hash value */ return 0; } It's easy enough to verify empirically. $ python -m timeit -s"s=''" "for i in xrange(10):s+='a'" 1000000 loops, best of 3: 1.85 usec per loop $ python -m timeit -s"s=''" "for i in xrange(100):s+='a'" 10000 loops, best of 3: 16.8 usec per loop $ python -m timeit -s"s=''" "for i in xrange(1000):s+='a'" 10000 loops, best of 3: 158 usec per loop $ python -m timeit -s"s=''" "for i in xrange(10000):s+='a'" 1000 loops, best of 3: 1.71 msec per loop $ python -m timeit -s"s=''" "for i in xrange(100000):s+='a'" 10 loops, best of 3: 14.6 msec per loop $ python -m timeit -s"s=''" "for i in xrange(1000000):s+='a'" 10 loops, best of 3: 173 msec per loop It's important however to note that this optimisation isn't part of the Python spec. It's only in the cPython implementation as far as I know. The same empirical testing on pypy or jython for example might show the older O(n**2) performance. $ pypy -m timeit -s"s=''" "for i in xrange(10):s+='a'" 10000 loops, best of 3: 90.8 usec per loop $ pypy -m timeit -s"s=''" "for i in xrange(100):s+='a'" 1000 loops, best of 3: 896 usec per loop $ pypy -m timeit -s"s=''" "for i in xrange(1000):s+='a'" 100 loops, best of 3: 9.03 msec per loop $ pypy -m timeit -s"s=''" "for i in xrange(10000):s+='a'" 10 loops, best of 3: 89.5 msec per loop So far so good, but then, $ pypy -m timeit -s"s=''" "for i in xrange(100000):s+='a'" 10 loops, best of 3: 12.8 sec per loop ouch even worse than quadratic. So pypy is doing something that works well with short strings, but performs poorly for larger strings.


Instruction:
Decorators with parameters?

I have a problem with the transfer of the variable insurance_mode by the decorator. I would do it by the following decorator statement: @execute_complete_reservation(True) def test_booking_gta_object(self): self.test_select_gta_object() but unfortunately, this statement does not work. Perhaps maybe there is better way to solve this problem. def execute_complete_reservation(test_case,insurance_mode): def inner_function(self,*args,**kwargs): self.test_create_qsf_query() test_case(self,*args,**kwargs) self.test_select_room_option() if insurance_mode: self.test_accept_insurance_crosseling() else: self.test_decline_insurance_crosseling() self.test_configure_pax_details() self.test_configure_payer_details return inner_function

Output: The syntax for decorators with arguments is a bit different - the decorator with arguments should return a function that will take a function and return another function. So it should really return a normal decorator. A bit confusing, right? What I mean is: def decorator_factory(argument): def decorator(function): def wrapper(*args, **kwargs): funny_stuff() something_with_argument(argument) result = function(*args, **kwargs) more_funny_stuff() return result return wrapper return decorator Here you can read more on the subject - it's also possible to implement this using callable objects and that is also explained there. Usage: @decorator_factory("Some argument") def function_to_be_decorated(args): print(f"Do something with '{args}'.") decorator_factory("Some argument") uses the given argument to create a standard, argumentless decorator. So the following block is functionally identical to the one above: created_decorator = decorator_factory("Some argument") @created_decorator def function_to_be_decorated(args): print(f"Do something with '{args}'.")


Instruction:
Correct way to try/except using Python requests module?

try: r = requests.get(url, params={'s': thing}) except requests.ConnectionError, e: print(e) Is this correct? Is there a better way to structure this? Will this cover all my bases?

Output: Have a look at the Requests exception docs. In short: In the event of a network problem (e.g. DNS failure, refused connection, etc), Requests will raise a ConnectionError exception. In the event of the rare invalid HTTP response, Requests will raise an HTTPError exception. If a request times out, a Timeout exception is raised. If a request exceeds the configured number of maximum redirections, a TooManyRedirects exception is raised. All exceptions that Requests explicitly raises inherit from requests.exceptions.RequestException. To answer your question, what you show will not cover all of your bases. You'll only catch connection-related errors, not ones that time out. What to do when you catch the exception is really up to the design of your script/program. Is it acceptable to exit? Can you go on and try again? If the error is catastrophic and you can't go on, then yes, you may abort your program by raising SystemExit (a nice way to both print an error and call sys.exit). You can either catch the base-class exception, which will handle all cases: try: r = requests.get(url, params={'s': thing}) except requests.exceptions.RequestException as e: # This is the correct syntax raise SystemExit(e) Or you can catch them separately and do different things. try: r = requests.get(url, params={'s': thing}) except requests.exceptions.Timeout: # Maybe set up for a retry, or continue in a retry loop except requests.exceptions.TooManyRedirects: # Tell the user their URL was bad and try a different one except requests.exceptions.RequestException as e: # catastrophic error. bail. raise SystemExit(e) As Christian pointed out: If you want http errors (e.g. 401 Unauthorized) to raise exceptions, you can call Response.raise_for_status. That will raise an HTTPError, if the response was an http error. An example: try: r = requests.get('http://www.google.com/nothere') r.raise_for_status() except requests.exceptions.HTTPError as err: raise SystemExit(err) Will print: 404 Client Error: Not Found for url: http://www.google.com/nothere


Instruction:
How to rename a file using Python

I want to change a.txt to b.kml.

Output: Use os.rename: import os os.rename('a.txt', 'b.kml') Usage: os.rename('from.extension.whatever','to.another.extension')


Instruction:
Is arr.__len__() the preferred way to get the length of an array in Python?

In Python, is the following the only way to get the number of elements? arr.__len__() If so, why the strange syntax?

Output: my_list = [1,2,3,4,5] len(my_list) # 5 The same works for tuples: my_tuple = (1,2,3,4,5) len(my_tuple) # 5 And strings, which are really just arrays of characters: my_string = 'hello world' len(my_string) # 11 It was intentionally done this way so that lists, tuples and other container types or iterables didn't all need to explicitly implement a public .length() method, instead you can just check the len() of anything that implements the 'magic' __len__() method. Sure, this may seem redundant, but length checking implementations can vary considerably, even within the same language. It's not uncommon to see one collection type use a .length() method while another type uses a .length property, while yet another uses .count(). Having a language-level keyword unifies the entry point for all these types. So even objects you may not consider to be lists of elements could still be length-checked. This includes strings, queues, trees, etc. The functional nature of len() also lends itself well to functional styles of programming. lengths = map(len, list_of_containers)


Instruction:
How can I check if a string represents an int, without using try/except?

Is there any way to tell whether a string represents an integer (e.g., '3', '-17' but not '3.14' or 'asfasfas') Without using a try/except mechanism? is_int('3.14') == False is_int('-7') == True

Output: If you're really just annoyed at using try/excepts all over the place, please just write a helper function: def represents_int(s): try: int(s) except ValueError: return False else: return True >>> print(represents_int("+123")) True >>> print(represents_int("10.0")) False It's going to be WAY more code to exactly cover all the strings that Python considers integers. I say just be pythonic on this one.


Instruction:
Difference between map, applymap and apply methods in Pandas

Can you tell me when to use these vectorization methods with basic examples? I see that map is a Series method whereas the rest are DataFrame methods. I got confused about apply and applymap methods though. Why do we have two methods for applying a function to a DataFrame? Again, simple examples which illustrate the usage would be great!

Output: Comparing map, applymap and apply: Context Matters The major differences are: Definition map is defined on Series only applymap is defined on DataFrames only apply is defined on both Input argument map accepts dict, Series, or callable applymap and apply accept callable only Behavior map is elementwise for Series applymap is elementwise for DataFrames apply also works elementwise but is suited to more complex operations and aggregation. The behaviour and return value depends on the function. Use case (the most important difference) map is meant for mapping values from one domain to another, so is optimised for performance, e.g., df['A'].map({1:'a', 2:'b', 3:'c'}) applymap is good for elementwise transformations across multiple rows/columns, e.g., df[['A', 'B', 'C']].applymap(str.strip) apply is for applying any function that cannot be vectorised, e.g., df['sentences'].apply(nltk.sent_tokenize) Also see When should I (not) want to use pandas apply() in my code? for a writeup I made a while back on the most appropriate scenarios for using apply. (Note that there aren't many, but there are a few— apply is generally slow.) Summarising map applymap apply Defined on Series? Yes No Yes Defined on DataFrame? No Yes Yes Argument dict, Series, or callable1 callable2 callable Elementwise? Yes Yes Yes Aggregation? No No Yes Use Case Transformation/mapping3 Transformation More complex functions Returns Series DataFrame scalar, Series, or DataFrame4 Footnotes map when passed a dictionary/Series will map elements based on the keys in that dictionary/Series. Missing values will be recorded as NaN in the output. applymap in more recent versions has been optimised for some operations. You will find applymap slightly faster than apply in some cases. My suggestion is to test them both and use whatever works better. map is optimised for elementwise mappings and transformation. Operations that involve dictionaries or Series will enable pandas to use faster code paths for better performance. Series.apply returns a scalar for aggregating operations, Series otherwise. Similarly for DataFrame.apply. Note that apply also has fastpaths when called with certain NumPy functions such as mean, sum, etc.


Instruction:
In practice, what are the main uses for the "yield from" syntax in Python 3.3?

I'm having a hard time wrapping my brain around PEP 380. What are the situations where yield from is useful? What is the classic use case? Why is it compared to micro-threads? So far I have used generators, but never really used coroutines (introduced by PEP-342). Despite some similarities, generators and coroutines are basically two different concepts. Understanding coroutines (not only generators) is the key to understanding the new syntax. IMHO coroutines are the most obscure Python feature, most books make it look useless and uninteresting. Thanks for the great answers, but special thanks to agf and his comment linking to David Beazley presentations.

Output: Let's get one thing out of the way first. The explanation that yield from g is equivalent to for v in g: yield v does not even begin to do justice to what yield from is all about. Because, let's face it, if all yield from does is expand the for loop, then it does not warrant adding yield from to the language and preclude a whole bunch of new features from being implemented in Python 2.x. What yield from does is it establishes a transparent, bidirectional connection between the caller and the sub-generator: The connection is "transparent" in the sense that it will propagate everything correctly, not just the elements being generated (e.g. exceptions are propagated). The connection is "bidirectional" in the sense that data can be both sent from and to a generator. (If we were talking about TCP, yield from g might mean "now temporarily disconnect my client's socket and reconnect it to this other server socket".) BTW, if you are not sure what sending data to a generator even means, you need to drop everything and read about coroutines first—they're very useful (contrast them with subroutines), but unfortunately lesser-known in Python. Dave Beazley's Curious Course on Coroutines is an excellent start. Read slides 24-33 for a quick primer. Reading data from a generator using yield from def reader(): """A generator that fakes a read from a file, socket, etc.""" for i in range(4): yield '<< %s' % i def reader_wrapper(g): # Manually iterate over data produced by reader for v in g: yield v wrap = reader_wrapper(reader()) for i in wrap: print(i) # Result << 0 << 1 << 2 << 3 Instead of manually iterating over reader(), we can just yield from it. def reader_wrapper(g): yield from g That works, and we eliminated one line of code. And probably the intent is a little bit clearer (or not). But nothing life changing. Sending data to a generator (coroutine) using yield from - Part 1 Now let's do something more interesting. Let's create a coroutine called writer that accepts data sent to it and writes to a socket, fd, etc. def writer(): """A coroutine that writes data *sent* to it to fd, socket, etc.""" while True: w = (yield) print('>> ', w) Now the question is, how should the wrapper function handle sending data to the writer, so that any data that is sent to the wrapper is transparently sent to the writer()? def writer_wrapper(coro): # TBD pass w = writer() wrap = writer_wrapper(w) wrap.send(None) # "prime" the coroutine for i in range(4): wrap.send(i) # Expected result >> 0 >> 1 >> 2 >> 3 The wrapper needs to accept the data that is sent to it (obviously) and should also handle the StopIteration when the for loop is exhausted. Evidently just doing for x in coro: yield x won't do. Here is a version that works. def writer_wrapper(coro): coro.send(None) # prime the coro while True: try: x = (yield) # Capture the value that's sent coro.send(x) # and pass it to the writer except StopIteration: pass Or, we could do this. def writer_wrapper(coro): yield from coro That saves 6 lines of code, make it much much more readable and it just works. Magic! Sending data to a generator yield from - Part 2 - Exception handling Let's make it more complicated. What if our writer needs to handle exceptions? Let's say the writer handles a SpamException and it prints *** if it encounters one. class SpamException(Exception): pass def writer(): while True: try: w = (yield) except SpamException: print('***') else: print('>> ', w) What if we don't change writer_wrapper? Does it work? Let's try # writer_wrapper same as above w = writer() wrap = writer_wrapper(w) wrap.send(None) # "prime" the coroutine for i in [0, 1, 2, 'spam', 4]: if i == 'spam': wrap.throw(SpamException) else: wrap.send(i) # Expected Result >> 0 >> 1 >> 2 *** >> 4 # Actual Result >> 0 >> 1 >> 2 Traceback (most recent call last): ... redacted ... File ... in writer_wrapper x = (yield) __main__.SpamException Um, it's not working because x = (yield) just raises the exception and everything comes to a crashing halt. Let's make it work, but manually handling exceptions and sending them or throwing them into the sub-generator (writer) def writer_wrapper(coro): """Works. Manually catches exceptions and throws them""" coro.send(None) # prime the coro while True: try: try: x = (yield) except Exception as e: # This catches the SpamException coro.throw(e) else: coro.send(x) except StopIteration: pass This works. # Result >> 0 >> 1 >> 2 *** >> 4 But so does this! def writer_wrapper(coro): yield from coro The yield from transparently handles sending the values or throwing values into the sub-generator. This still does not cover all the corner cases though. What happens if the outer generator is closed? What about the case when the sub-generator returns a value (yes, in Python 3.3+, generators can return values), how should the return value be propagated? That yield from transparently handles all the corner cases is really impressive. yield from just magically works and handles all those cases. I personally feel yield from is a poor keyword choice because it does not make the two-way nature apparent. There were other keywords proposed (like delegate but were rejected because adding a new keyword to the language is much more difficult than combining existing ones. In summary, it's best to think of yield from as a transparent two way channel between the caller and the sub-generator. References: PEP 380 - Syntax for delegating to a sub-generator (Ewing) [v3.3, 2009-02-13] PEP 342 - Coroutines via Enhanced Generators (GvR, Eby) [v2.5, 2005-05-10]


Instruction:
Dealing with multiple Python versions and PIP

Is there a way to make pip play well with multiple versions of Python? For example, I want to use pip to explicitly install things to either my site 2.5 installation or my site 2.6 installation. For example, with easy_install, I use easy_install-2.{5,6}. And, yes — I know about virtualenv, and no — it's not a solution to this particular problem.

Output: The current recommendation is to use python -m pip, where python is the version of Python you would like to use. This is the recommendation because it works across all versions of Python, and in all forms of virtualenv. For example: # The system default Python installation: python -m pip install fish # A virtualenv's Python installation: .env/bin/python -m pip install fish # A specific version of python: python-3.6 -m pip install fish Previous answer, left for posterity: Since version 0.8, Pip supports pip-{version}. You can use it the same as easy_install-{version}: pip-2.5 install myfoopackage pip-2.6 install otherpackage pip-2.7 install mybarpackage pip changed its schema to use pipVERSION instead of pip-VERSION in version 1.5. You should use the following if you have pip >= 1.5: pip2.6 install otherpackage pip2.7 install mybarpackage Check Versioned commands consistent with Python. #1053 for more details References: Need pip-x.y scripts #200 v0.8 changelog or News for pip, v0.8


Instruction:
What is the maximum recursion depth, and how to increase it?

I have this tail recursive function here: def recursive_function(n, sum): if n < 1: return sum else: return recursive_function(n-1, sum+n) c = 998 print(recursive_function(c, 0)) It works up to n=997, then it just breaks and spits out a RecursionError: maximum recursion depth exceeded in comparison. Is this just a stack overflow? Is there a way to get around it?

Output: It is a guard against a stack overflow, yes. Python (or rather, the CPython implementation) doesn't optimize tail recursion, and unbridled recursion causes stack overflows. You can check the recursion limit with sys.getrecursionlimit: import sys print(sys.getrecursionlimit()) and change the recursion limit with sys.setrecursionlimit: sys.setrecursionlimit(1500) but doing so is dangerous -- the standard limit is a little conservative, but Python stackframes can be quite big. Python isn't a functional language and tail recursion is not a particularly efficient technique. Rewriting the algorithm iteratively, if possible, is generally a better idea.


Instruction:
CSV file written with Python has blank lines between each row

import csv with open('thefile.csv', 'rb') as f: data = list(csv.reader(f)) import collections counter = collections.defaultdict(int) for row in data: counter[row[10]] += 1 with open('/pythonwork/thefile_subset11.csv', 'w') as outfile: writer = csv.writer(outfile) for row in data: if counter[row[10]] >= 504: writer.writerow(row) This code reads thefile.csv, makes changes, and writes results to thefile_subset1. However, when I open the resulting csv in Microsoft Excel, there is an extra blank line after each record! Is there a way to make it not put an extra blank line?

Output: The csv.writer module directly controls line endings and writes \r\n into the file directly. In Python 3 the file must be opened in untranslated text mode with the parameters 'w', newline='' (empty string) or it will write \r\r\n on Windows, where the default text mode will translate each \n into \r\n. #!python3 with open('/pythonwork/thefile_subset11.csv', 'w', newline='') as outfile: writer = csv.writer(outfile) If using the Path module: from pathlib import Path import csv with Path('/pythonwork/thefile_subset11.csv').open('w', newline='') as outfile: writer = csv.writer(outfile) If using the StringIO module to build an in-memory result, the result string will contain the translated line terminator: from io import StringIO import csv s = StringIO() writer = csv.writer(s) writer.writerow([1,2,3]) print(repr(s.getvalue())) # '1,2,3\r\n' (Windows result) If writing that string to a file later, remember to use newline='': # built-in open() with open('/pythonwork/thefile_subset11.csv', 'w', newline='') as f: f.write(s.getvalue()) # Path's open() with Path('/pythonwork/thefile_subset11.csv').open('w', newline='') as f: f.write(s.getvalue()) # Path's write_text() added the newline parameter to Python 3.10. Path('/pythonwork/thefile_subset11.csv').write_text(s.getvalue(), newline='') In Python 2, use binary mode to open outfile with mode 'wb' instead of 'w' to prevent Windows newline translation. Python 2 also has problems with Unicode and requires other workarounds to write non-ASCII text. See the Python 2 link below and the UnicodeReader and UnicodeWriter examples at the end of the page if you have to deal with writing Unicode strings to CSVs on Python 2, or look into the 3rd party unicodecsv module: #!python2 with open('/pythonwork/thefile_subset11.csv', 'wb') as outfile: writer = csv.writer(outfile) Documentation Links https://docs.python.org/3/library/csv.html#csv.writer https://docs.python.org/2/library/csv.html#csv.writer


Instruction:
Finding local IP addresses using Python's stdlib

How can I find local IP addresses (i.e. 192.168.x.x or 10.0.x.x) in Python platform independently and using only the standard library?

Output: import socket socket.gethostbyname(socket.gethostname()) This won't work always (returns 127.0.0.1 on machines having the hostname in /etc/hosts as 127.0.0.1), a paliative would be what gimel shows, use socket.getfqdn() instead. Of course your machine needs a resolvable hostname. If you need to get all the ip addresses, you can instead use: socket.gethostbyname_ex(socket.gethostname()) See the documentation for more information about gethostbyname_ex.


Instruction:
How to install psycopg2 with "pip" on Python?

I'm using virtualenv and I need to install "psycopg2". I have done the following: pip install http://pypi.python.org/packages/source/p/psycopg2/psycopg2-2.4.tar.gz#md5=24f4368e2cfdc1a2b03282ddda814160 And I have the following messages: Downloading/unpacking http://pypi.python.org/packages/source/p/psycopg2/psycopg2 -2.4.tar.gz#md5=24f4368e2cfdc1a2b03282ddda814160 Downloading psycopg2-2.4.tar.gz (607Kb): 607Kb downloaded Running setup.py egg_info for package from http://pypi.python.org/packages/sou rce/p/psycopg2/psycopg2-2.4.tar.gz#md5=24f4368e2cfdc1a2b03282ddda814160 Error: pg_config executable not found. Please add the directory containing pg_config to the PATH or specify the full executable path with the option: python setup.py build_ext --pg-config /path/to/pg_config build ... or with the pg_config option in 'setup.cfg'. Complete output from command python setup.py egg_info: running egg_info creating pip-egg-info\psycopg2.egg-info writing pip-egg-info\psycopg2.egg-info\PKG-INFO writing top-level names to pip-egg-info\psycopg2.egg-info\top_level.txt writing dependency_links to pip-egg-info\psycopg2.egg-info\dependency_links.txt writing manifest file 'pip-egg-info\psycopg2.egg-info\SOURCES.txt' warning: manifest_maker: standard file '-c' not found Error: pg_config executable not found. Please add the directory containing pg_config to the PATH or specify the full executable path with the option: python setup.py build_ext --pg-config /path/to/pg_config build ... or with the pg_config option in 'setup.cfg'. ---------------------------------------- Command python setup.py egg_info failed with error code 1 Storing complete log in C:\Documents and Settings\anlopes\Application Data\pip\p ip.log My question, I only need to do this to get the psycopg2 working? python setup.py build_ext --pg-config /path/to/pg_config build ...

Output: Note: Since a while back, there are binary wheels for Windows in PyPI, so this should no longer be an issue for Windows users. Below are solutions for Linux, Mac users, since lots of them find this post through web searches. Option 1 Install the psycopg2-binary PyPI package instead, it has Python wheels for Linux and Mac OS. pip install psycopg2-binary Option 2 Install the prerequsisites for building the psycopg2 package from source: Debian/Ubuntu Python version Command Note Default Python 3 sudo apt install libpq-dev python3-dev Python 3.x sudo apt install libpq-dev python3.x-dev substitute x in command Python 2 sudo apt install libpq-dev python-dev If that's not enough, you might additionally need to install sudo apt install build-essential or sudo apt install postgresql-server-dev-all as well before installing psycopg2 again. CentOS 6 See Banjer's answer macOS See nichochar's answer


Instruction:
How to set the current working directory?

How to set the current working directory in Python?

Output: Try os.chdir import os os.chdir(path) Change the current working directory to path. Availability: Unix, Windows.


Instruction:
Getting the index of the returned max or min item using max()/min() on a list

I'm using Python's max and min functions on lists for a minimax algorithm, and I need the index of the value returned by max() or min(). In other words, I need to know which move produced the max (at a first player's turn) or min (second player) value. for i in range(9): new_board = current_board.new_board_with_move([i / 3, i % 3], player) if new_board: temp = min_max(new_board, depth + 1, not is_min_level) values.append(temp) if is_min_level: return min(values) else: return max(values) I need to be able to return the actual index of the min or max value, not just the value.

Output: Find the minimum value with min() then find that value's index with .index(): values.index(min(values)) Or the maximum: values.index(max(values)) If your list contains repeats of the minimum or maximum value this will return the index of the first one.


Instruction:
How do I find the duplicates in a list and create another list with them?

How do I find the duplicates in a list of integers and create another list of the duplicates?

Output: To remove duplicates use set(a). To print duplicates, something like: a = [1,2,3,2,1,5,6,5,5,5] import collections print([item for item, count in collections.Counter(a).items() if count > 1]) ## [1, 2, 5] Note that Counter is not particularly efficient (timings) and probably overkill here. set will perform better. This code computes a list of unique elements in the source order: seen = set() uniq = [] for x in a: if x not in seen: uniq.append(x) seen.add(x) or, more concisely: seen = set() uniq = [x for x in a if x not in seen and not seen.add(x)] I don't recommend the latter style, because it is not obvious what not seen.add(x) is doing (the set add() method always returns None, hence the need for not). To compute the list of duplicated elements without libraries: seen = set() dupes = set() for x in a: if x in seen: dupes.add(x) else: seen.add(x) or, more concisely: seen = set() dupes = {x for x in a if x in seen or seen.add(x)} If list elements are not hashable, you cannot use sets/dicts and have to resort to a quadratic time solution (compare each with each). For example: a = [[1], [2], [3], [1], [5], [3]] no_dupes = [x for n, x in enumerate(a) if x not in a[:n]] print no_dupes # [[1], [2], [3], [5]] dupes = [x for n, x in enumerate(a) if x in a[:n]] print dupes # [[1], [3]]


Instruction:
How do I capture SIGINT in Python?

I'm working on a python script that starts several processes and database connections. Every now and then I want to kill the script with a Ctrl+C signal, and I'd like to do some cleanup. In Perl I'd do this: $SIG{'INT'} = 'exit_gracefully'; sub exit_gracefully { print "Caught ^C \n"; exit (0); } How do I do the analogue of this in Python?

Output: Register your handler with signal.signal like this: #!/usr/bin/env python import signal import sys def signal_handler(sig, frame): print('You pressed Ctrl+C!') sys.exit(0) signal.signal(signal.SIGINT, signal_handler) print('Press Ctrl+C') signal.pause() Code adapted from here. More documentation on signal can be found here.


Instruction:
How can I import a module dynamically given its name as string?

I'm writing a Python application that takes a command as an argument, for example: $ python myapp.py command1 I want the application to be extensible, that is, to be able to add new modules that implement new commands without having to change the main application source. The tree looks something like: myapp/ __init__.py commands/ __init__.py command1.py command2.py foo.py bar.py So I want the application to find the available command modules at runtime and execute the appropriate one. Python defines an __import__() function, which takes a string for a module name: __import__(name, globals=None, locals=None, fromlist=(), level=0) The function imports the module name, potentially using the given globals and locals to determine how to interpret the name in a package context. The fromlist gives the names of objects or submodules that should be imported from the module given by name. Source: https://docs.python.org/3/library/functions.html#__import__ So currently I have something like: command = sys.argv[1] try: command_module = __import__("myapp.commands.%s" % command, fromlist=["myapp.commands"]) except ImportError: # Display error message command_module.run() This works just fine, I'm just wondering if there is possibly a more idiomatic way to accomplish what we are doing with this code. Note that I specifically don't want to get in to using eggs or extension points. This is not an open-source project and I don't expect there to be "plugins". The point is to simplify the main application code and remove the need to modify it each time a new command module is added. See also: How can I import a module dynamically given the full path?

Output: With Python older than 2.7/3.1, that's pretty much how you do it. For newer versions, see importlib.import_module for Python 2 and Python 3. Or using __import__ you can import a list of modules by doing this: >>> moduleNames = ['sys', 'os', 're', 'unittest'] >>> moduleNames ['sys', 'os', 're', 'unittest'] >>> modules = map(__import__, moduleNames) Ripped straight from Dive Into Python.


Instruction:
How do I add default parameters to functions when using type hinting?

If I have a function like this: def foo(name, opts={}): pass And I want to add type hints to the parameters, how do I do it? The way I assumed gives me a syntax error: def foo(name: str, opts={}: dict) -> str: pass The following doesn't throw a syntax error but it doesn't seem like the intuitive way to handle this case: def foo(name: str, opts: dict={}) -> str: pass I can't find anything in the typing documentation or on a Google search. Edit: I didn't know how default arguments worked in Python, but for the sake of this question, I will keep the examples above. In general it's much better to do the following: def foo(name: str, opts: dict=None) -> str: if not opts: opts={} pass

Output: Your second way is correct. def foo(opts: dict = {}): pass print(foo.__annotations__) this outputs {'opts': <class 'dict'>} Although it is not explicitly mentioned in PEP 484, type hints are a specific use of function annotations, as outlined in PEP 3107. The syntax section clearly demonstrates that keyword arguments can be annotated in this manner. I strongly advise against using mutable keyword arguments. More information here.


Instruction:
How to specify multiple types using type-hints

I have a function in python that can either return a bool or a list. Is there a way to specify the types using type hints? For example, is this the correct way to do it? def foo(id) -> list or bool: ...

Output: From the documentation - Union Type: A union object holds the value of the | (bitwise or) operation on multiple type objects. These types are intended primarily for type annotations. The union type expression enables cleaner type hinting syntax compared to typing.Union. This use of | was added in Python 3.10. Hence the proper way to represent more than one return data type is: def foo(client_id: str) -> list | bool: For earlier versions, use typing.Union: from typing import Union def foo(client_id: str) -> Union[list, bool]: But do note that typing is not enforced. Python continues to remain a dynamically-typed language. The annotation syntax has been developed to help during the development of the code prior to being released into production. As PEP 484 states, "no type checking happens at runtime." >>> def foo(a: str) -> list: ... return "Works" ... >>> foo(1) 'Works' As you can see I am passing an int value and returning a str. However the __annotations__ will be set to the respective values. >>> foo.__annotations__ {'return': <class 'list'>, 'a': <class 'str'>} Please go through PEP 483 for more about Type hints. Also see What are type hints in Python 3.5?? Kindly note that this is available only for Python 3.5 and upwards. This is mentioned clearly in PEP 484.


Instruction:
What is the python keyword "with" used for?

What is the python keyword "with" used for? Example from: http://docs.python.org/tutorial/inputoutput.html >>> with open('/tmp/workfile', 'r') as f: ... read_data = f.read() >>> f.closed True

Output: In python the with keyword is used when working with unmanaged resources (like file streams). It is similar to the using statement in VB.NET and C#. It allows you to ensure that a resource is "cleaned up" when the code that uses it finishes running, even if exceptions are thrown. It provides 'syntactic sugar' for try/finally blocks. From Python Docs: The with statement clarifies code that previously would use try...finally blocks to ensure that clean-up code is executed. The with statement is a control-flow structure whose basic structure is: with expression [as variable]: with-block The expression is evaluated, and it should result in an object that supports the context management protocol (that is, has __enter__() and __exit__() methods). Update fixed VB callout per Scott Wisniewski's comment. I was indeed confusing with with using.


Instruction:
pip uses incorrect cached package version, instead of the user-specified version

I need to install psycopg2 v2.4.1 specifically. I accidentally did: pip install psycopg2 Instead of: pip install psycopg2==2.4.1 That installs 2.4.4 instead of the earlier version. Now even after I pip uninstall psycopg2 and attempt to reinstall with the correct version, it appears that pip is re-using the cache it downloaded the first time. How can I force pip to clear out its download cache and use the specific version I'm including in the command?

Output: If using pip 6.0 or newer, try adding the --no-cache-dir option (source). If using pip older than pip 6.0, upgrade it with pip install -U pip.


Instruction:
How do I check if a pandas DataFrame is empty?

How do I check if a pandas DataFrame is empty? I'd like to print some message in the terminal if the DataFrame is empty.

Output: You can use the attribute df.empty to check whether it's empty or not: if df.empty: print('DataFrame is empty!') Source: Pandas Documentation


Instruction:
How to iterate over a list in chunks

I have a Python script which takes as input a list of integers, which I need to work with four integers at a time. Unfortunately, I don't have control of the input, or I'd have it passed in as a list of four-element tuples. Currently, I'm iterating over it this way: for i in range(0, len(ints), 4): # dummy op for example code foo += ints[i] * ints[i + 1] + ints[i + 2] * ints[i + 3] It looks a lot like "C-think", though, which makes me suspect there's a more pythonic way of dealing with this situation. The list is discarded after iterating, so it needn't be preserved. Perhaps something like this would be better? while ints: foo += ints[0] * ints[1] + ints[2] * ints[3] ints[0:4] = [] Still doesn't quite "feel" right, though. :-/ Update: With the release of Python 3.12, I've changed the accepted answer. For anyone who has not (or cannot) make the jump to Python 3.12 yet, I encourage you to check out the previous accepted answer or any of the other excellent, backwards-compatible answers below. Related question: How do you split a list into evenly sized chunks in Python?

Output: As of Python 3.12, the itertools module gains a batched function that specifically covers iterating over batches of an input iterable, where the final batch may be incomplete (each batch is a tuple). Per the example code given in the docs: >>> for batch in batched('ABCDEFG', 3): ... print(batch) ... ('A', 'B', 'C') ('D', 'E', 'F') ('G',) Performance notes: The implementation of batched, like all itertools functions to date, is at the C layer, so it's capable of optimizations Python level code cannot match, e.g. On each pull of a new batch, it proactively allocates a tuple of precisely the correct size (for all but the last batch), instead of building the tuple up element by element with amortized growth causing multiple reallocations (the way a solution calling tuple on an islice does) It only needs to look up the .__next__ function of the underlying iterator once per batch, not n times per batch (the way a zip_longest((iter(iterable),) * n)-based approach does) The check for the end case is a simple C level NULL check (trivial, and required to handle possible exceptions anyway) Handling the end case is a C goto followed by a direct realloc (no making a copy into a smaller tuple) down to the already known final size, since it's tracking how many elements it has successfully pulled (no complex "create sentinel for use as fillvalue and do Python level if/else checks for each batch to see if it's empty, with the final batch requiring a search for where the fillvalue appeared last, to create the cut-down tuple" required by zip_longest-based solutions). Between all these advantages, it should massively outperform any Python-level solution (even highly optimized ones that push most or all of the per-item work to the C layer), regardless of whether the input iterable is long or short, and regardless of whether the batch size and the size of the final (possibly incomplete) batch (zip_longest-based solutions using guaranteed unique fillvalues for safety are the best possible solution for almost all cases when itertools.batched is not available, but they can suffer in pathological cases of "few large batches, with final batch mostly, not completely, filled", especially pre-3.10 when bisect can't be used to optimize slicing off the fillvalues from O(n) linear search down to O(log n) binary search, but batched avoids that search entirely, so it won't experience pathological cases at all).


Instruction:
How do I use itertools.groupby()?

I haven't been able to find an understandable explanation of how to actually use Python's itertools.groupby() function. What I'm trying to do is this: Take a list - in this case, the children of an objectified lxml element Divide it into groups based on some criteria Then later iterate over each of these groups separately. I've reviewed the documentation, but I've had trouble trying to apply them beyond a simple list of numbers. So, how do I use of itertools.groupby()? Is there another technique I should be using? Pointers to good "prerequisite" reading would also be appreciated.

Output: IMPORTANT NOTE: You may have to sort your data first. The part I didn't get is that in the example construction groups = [] uniquekeys = [] for k, g in groupby(data, keyfunc): groups.append(list(g)) # Store group iterator as a list uniquekeys.append(k) k is the current grouping key, and g is an iterator that you can use to iterate over the group defined by that grouping key. In other words, the groupby iterator itself returns iterators. Here's an example of that, using clearer variable names: from itertools import groupby things = [("animal", "bear"), ("animal", "duck"), ("plant", "cactus"), ("vehicle", "speed boat"), ("vehicle", "school bus")] for key, group in groupby(things, lambda x: x[0]): for thing in group: print("A %s is a %s." % (thing[1], key)) print("") This will give you the output: A bear is a animal. A duck is a animal. A cactus is a plant. A speed boat is a vehicle. A school bus is a vehicle. In this example, things is a list of tuples where the first item in each tuple is the group the second item belongs to. The groupby() function takes two arguments: (1) the data to group and (2) the function to group it with. Here, lambda x: x[0] tells groupby() to use the first item in each tuple as the grouping key. In the above for statement, groupby returns three (key, group iterator) pairs - once for each unique key. You can use the returned iterator to iterate over each individual item in that group. Here's a slightly different example with the same data, using a list comprehension: for key, group in groupby(things, lambda x: x[0]): listOfThings = " and ".join([thing[1] for thing in group]) print(key + "s: " + listOfThings + ".") This will give you the output: animals: bear and duck. plants: cactus. vehicles: speed boat and school bus.


Instruction:
pandas.parser.CParserError: Error tokenizing data

I'm trying to use pandas to manipulate a .csv file but I get this error: pandas.parser.CParserError: Error tokenizing data. C error: Expected 2 fields in line 3, saw 12 I have tried to read the pandas docs, but found nothing. My code is simple: path = 'GOOG Key Ratios.csv' #print(open(path).read()) data = pd.read_csv(path) How can I resolve this? Should I use the csv module or another language?

Output: you could also try; data = pd.read_csv('file1.csv', on_bad_lines='skip') Do note that this will cause the offending lines to be skipped. If you don't expect many bad lines and want to (at least) know their amount and IDs, use on_bad_lines='warn'. For advanced handling of bads, you can pass a callable. Edit For Pandas < 1.3.0 try data = pd.read_csv("file1.csv", error_bad_lines=False) as per pandas API reference.


Instruction:
if else in a list comprehension

I have a list l: l = [22, 13, 45, 50, 98, 69, 43, 44, 1] For numbers above 45 inclusive, I would like to add 1; and for numbers less than it, 5. I tried [x+1 for x in l if x >= 45 else x+5] But it gives me a syntax error. How can I achieve an if – else like this in a list comprehension?

Output: >>> l = [22, 13, 45, 50, 98, 69, 43, 44, 1] >>> [x+1 if x >= 45 else x+5 for x in l] [27, 18, 46, 51, 99, 70, 48, 49, 6] Do-something if <condition>, else do-something else.


Instruction:
What is a Python egg?

I'm trying to understand how Python packages work. Presumably eggs are some sort of packaging mechanism, but what would be a quick overview of what role they play and may be some information on why they're useful and how to create them?

Output: *Note: Egg packaging has been superseded by Wheel packaging.* Same concept as a .jar file in Java, it is a .zip file with some metadata files renamed .egg, for distributing code as bundles. Specifically: The Internal Structure of Python Eggs A "Python egg" is a logical structure embodying the release of a specific version of a Python project, comprising its code, resources, and metadata. There are multiple formats that can be used to physically encode a Python egg, and others can be developed. However, a key principle of Python eggs is that they should be discoverable and importable. That is, it should be possible for a Python application to easily and efficiently find out what eggs are present on a system, and to ensure that the desired eggs' contents are importable. The .egg format is well-suited to distribution and the easy uninstallation or upgrades of code, since the project is essentially self-contained within a single directory or file, unmingled with any other projects' code or resources. It also makes it possible to have multiple versions of a project simultaneously installed, such that individual programs can select the versions they wish to use.


Instruction:
Return JSON response from Flask view

I have a function that analyzes a CSV file with Pandas and produces a dict with summary information. I want to return the results as a response from a Flask view. How do I return a JSON response? @app.route("/summary") def summary(): d = make_summary() # send it back as json

Output: A view can directly return a Python dict or list and Flask will call jsonify automatically. @app.route("/summary") def summary(): d = make_summary() return d For older Flask versions, or to return a different JSON-serializable object, import and use jsonify. from flask import jsonify @app.route("/summary") def summary(): d = make_summary() return jsonify(d)


Instruction:
Is it possible to break a long line to multiple lines in Python?

Just like C, you can break a long line into multiple short lines. But in Python, if I do this, there will be an indent error... Is it possible?

Output: From PEP 8 - Style Guide for Python Code: The preferred way of wrapping long lines is by using Python's implied line continuation inside parentheses, brackets and braces. If necessary, you can add an extra pair of parentheses around an expression, but sometimes using a backslash looks better. Make sure to indent the continued line appropriately. Example of implicit line continuation: a = ( '1' + '2' + '3' - '4' ) b = some_function( param1=foo( "a", "b", "c" ), param2=bar("d"), ) On the topic of line breaks around a binary operator, it goes on to say: For decades the recommended style was to break after binary operators. But this can hurt readability in two ways: the operators tend to get scattered across different columns on the screen, and each operator is moved away from its operand and onto the previous line. In Python code, it is permissible to break before or after a binary operator, as long as the convention is consistent locally. For new code Knuth's style (line breaks before the operator) is suggested. Example of explicit line continuation: a = '1' \ + '2' \ + '3' \ - '4'


Instruction:
What is related_name used for?

What is the related_name argument useful for on ManyToManyField and ForeignKey fields? For example, given the following code, what is the effect of related_name='maps'? class Map(db.Model): members = models.ManyToManyField(User, related_name='maps', verbose_name=_('members'))

Output: The related_name attribute specifies the name of the reverse relation from the User model back to your model. If you don't specify a related_name, Django automatically creates one using the name of your model with the suffix _set, for instance User.map_set.all(). If you do specify, e.g. related_name=maps on the User model, User.map_set will still work, but the User.maps. syntax is obviously a bit cleaner and less clunky; so for example, if you had a user object current_user, you could use current_user.maps.all() to get all instances of your Map model that have a relation to current_user. The Django documentation has more details. To disable creating the backwards relationship entirely, set related_name to "+".


Instruction:
Disable output buffering

Is output buffering enabled by default in Python's interpreter for sys.stdout? If the answer is positive, what are all the ways to disable it? Suggestions so far: Use the -u command line switch Wrap sys.stdout in an object that flushes after every write Set PYTHONUNBUFFERED env var sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0) Is there any other way to set some global flag in sys/sys.stdout programmatically during execution? If you just want to flush after a specific write using print, see How can I flush the output of the print function?.

Output: From Magnus Lycka answer on a mailing list: You can skip buffering for a whole python process using python -u or by setting the environment variable PYTHONUNBUFFERED. You could also replace sys.stdout with some other stream like wrapper which does a flush after every call. class Unbuffered(object): def __init__(self, stream): self.stream = stream def write(self, data): self.stream.write(data) self.stream.flush() def writelines(self, datas): self.stream.writelines(datas) self.stream.flush() def __getattr__(self, attr): return getattr(self.stream, attr) import sys sys.stdout = Unbuffered(sys.stdout) print 'Hello'


Instruction:
Does Python's time.time() return the local or UTC timestamp?

Does time.time() in the Python time module return the system's time or the time in UTC?

Output: The time.time() function returns the number of seconds since the epoch, as a float. Note that “the epoch” is defined as the start of January 1st, 1970 in UTC. So the epoch is defined in terms of UTC and establishes a global moment in time. No matter where on Earth you are, “seconds past epoch” (time.time()) returns the same value at the same moment. Here is some sample output I ran on my computer, converting it to a string as well. >>> import time >>> ts = time.time() >>> ts 1355563265.81 >>> import datetime >>> datetime.datetime.fromtimestamp(ts).strftime('%Y-%m-%d %H:%M:%S') '2012-12-15 01:21:05' >>> The ts variable is the time returned in seconds. I then converted it to a human-readable string using the datetime library.


Instruction:
Simple argparse example wanted: 1 argument, 3 results

The documentation for the argparse python module, while excellent I'm sure, is too much for my tiny beginner brain to grasp right now. All I want to do is "If arg is A, do this, if B do that, if none of the above show help and quit". I don't need to do math on the command line or meddle with formatting lines on the screen or change option characters.

Output: My understanding of the question is two-fold. First, the simplest possible argparse example. Of course, to be dead-simple, it's got to be a toy example, i.e. all overhead with little power, but it might get you started. import argparse parser = argparse.ArgumentParser() parser.add_argument("a") args = parser.parse_args() if args.a == 'magic.name': print('You nailed it!') But this positional argument is now required. If you leave it out when invoking this program, you'll get an error about missing arguments. This leads me to the second part of the question. You seem to want a single optional argument without a named label (the --option labels). My suggestion would be to modify the code above as follows: import argparse parser = argparse.ArgumentParser() parser.add_argument("a", nargs='?') args = parser.parse_args() if args.a is None: print('I can tell that no argument was given and I can deal with that here.') elif args.a == 'magic.name': print('You nailed it!') else: print(args.a) There may well be a more elegant solution, but this works and is minimalist. Note: If you want a different default value instead of None, use the default parameter to .add_argument.


Instruction:
How can I specify the function type in my type hints?

How can I specify the type hint of a variable as a function type? There is no typing.Function, and I could not find anything in the relevant PEP, PEP 483.

Output: As @jonrsharpe noted in a comment, this can be done with collections.abc.Callable: from collections.abc import Callable def my_function(func: Callable): Note: Callable on its own is equivalent to Callable[..., Any]. Such a Callable takes any number and type of arguments (...) and returns a value of any type (Any). If this is too unconstrained, one may also specify the types of the input argument list and return type. For example, given: def sum(a: int, b: int) -> int: return a+b The corresponding annotation is: Callable[[int, int], int] That is, the parameters are sub-scripted in the outer subscription with the return type as the second element in the outer subscription. In general: Callable[[ParamType1, ParamType2, ..., ParamTypeN], ReturnType]