Instruction:
How to remove multiple items from a list in just one statement?

In python, I know how to remove items from a list: item_list = ['item', 5, 'foo', 3.14, True] item_list.remove('item') item_list.remove(5) The above code removes the values 5 and 'item' from item_list. But when there is a lot of stuff to remove, I have to write many lines of: item_list.remove("something_to_remove") If I know the index of what I am removing, I use: del item_list[x] where x is the index of the item I want to remove. If I know the index of all of the numbers that I want to remove, I'll use some sort of loop to del the items at the indices. But what if I don't know the indices of the items I want to remove? I tried item_list.remove('item', 'foo'), but I got an error saying that remove only takes one argument. Is there a way to remove multiple items from a list in a single statement? P.S. I've used del and remove. Can someone explain the difference between these two, or are they the same?

Output: In Python, creating a new object e.g. with a list comprehension is often better than modifying an existing one: item_list = ['item', 5, 'foo', 3.14, True] item_list = [e for e in item_list if e not in ('item', 5)] ... which is equivalent to: item_list = ['item', 5, 'foo', 3.14, True] new_list = [] for e in item_list: if e not in ('item', 5): new_list.append(e) item_list = new_list In case of a big list of filtered out values (here, ('item', 5) is a small set of elements), using a set is faster as the in operation is O(1) time complexity on average. It's also a good idea to build the iterable you're removing first, so that you're not creating it on every iteration of the list comprehension: unwanted = {'item', 5} item_list = [e for e in item_list if e not in unwanted] A bloom filter is also a good solution if memory is not cheap.


Instruction:
What values are valid in Pandas 'Freq' tags?

I am trying to use date_range. I came across some values valid for freq, like BME and BMS and I would like to be able to quickly look up the proper strings to get what I want. What values are valid in Pandas 'Freq' tags?

Output: You can find it called Offset Aliases: A number of string aliases are given to useful common time series frequencies. We will refer to these aliases as offset aliases. Alias Description B business day frequency C custom business day frequency D calendar day frequency W weekly frequency ME month end frequency SME semi-month end frequency (15th and end of month) BME business month end frequency CBME custom business month end frequency MS month start frequency SMS semi-month start frequency (1st and 15th) BMS business month start frequency CBMS custom business month start frequency QE quarter end frequency BQE business quarter end frequency QS quarter start frequency BQS business quarter start frequency YE year end frequency BYE business year end frequency YS year start frequency BYS business year start frequency h hourly frequency bh business hour frequency cbh custom business hour frequency min minutely frequency s secondly frequency ms milliseconds us microseconds ns nanoseconds


Instruction:
Why does "a == x or y or z" always evaluate to True? How can I compare "a" to all of those?

I am writing a security system that denies access to unauthorized users. name = input("Hello. Please enter your name: ") if name == "Kevin" or "Jon" or "Inbar": print("Access granted.") else: print("Access denied.") It grants access to authorized users as expected, but it also lets in unauthorized users! Hello. Please enter your name: Bob Access granted. Why does this occur? I've plainly stated to only grant access when name equals Kevin, Jon, or Inbar. I have also tried the opposite logic, if "Kevin" or "Jon" or "Inbar" == name, but the result is the same. This question is intended as the canonical duplicate target of this very common problem. There is another popular question How to test multiple variables for equality against a single value? that has the same fundamental problem, but the comparison targets are reversed. This question should not be closed as a duplicate of that one as this problem is encountered by newcomers to Python who might have difficulties applying the knowledge from the reversed question to their problem. For in instead of ==, there are solutions here: How to test the membership of multiple values in a list

Output: In many cases, Python looks and behaves like natural English, but this is one case where that abstraction fails. People can use context clues to determine that "Jon" and "Inbar" are objects joined to the verb "equals", but the Python interpreter is more literal minded. if name == "Kevin" or "Jon" or "Inbar": is logically equivalent to: if (name == "Kevin") or ("Jon") or ("Inbar"): Which, for user Bob, is equivalent to: if (False) or ("Jon") or ("Inbar"): The or operator chooses the first operand that is "truthy", i.e. which would satisfy an if condition (or the last one, if none of them are "truthy"): if "Jon": Since "Jon" is truthy, the if block executes. That is what causes "Access granted" to be printed regardless of the name given. All of this reasoning also applies to the expression if "Kevin" or "Jon" or "Inbar" == name. the first value, "Kevin", is true, so the if block executes. There are three common ways to properly construct this conditional. Use multiple == operators to explicitly check against each value: if name == "Kevin" or name == "Jon" or name == "Inbar": Compose a collection of valid values (a set, a list or a tuple for example), and use the in operator to test for membership: if name in {"Kevin", "Jon", "Inbar"}: Use any() and a generator expression to explicitly check against each value in a loop: if any(name == auth for auth in ["Kevin", "Jon", "Inbar"]): In general the second should be preferred as it's easier to read and also faster: >>> import timeit >>> timeit.timeit('name == "Kevin" or name == "Jon" or name == "Inbar"', setup="name='Inbar'") 0.0960568820592016 >>> timeit.timeit('name in {"Kevin", "Jon", "Inbar"}', setup="name='Inbar'") 0.034957461059093475 >>> timeit.timeit('any(name == auth for auth in ["Kevin", "Jon", "Inbar"])', setup="name='Inbar'") 0.6511583919636905 For those who may want proof that if a == b or c or d or e: ... is indeed parsed like this. The built-in ast module provides an answer: >>> import ast >>> ast.parse("a == b or c or d or e", "<string>", "eval") <ast.Expression object at 0x7f929c898220> >>> print(ast.dump(_, indent=4)) Expression( body=BoolOp( op=Or(), values=[ Compare( left=Name(id='a', ctx=Load()), ops=[ Eq()], comparators=[ Name(id='b', ctx=Load())]), Name(id='c', ctx=Load()), Name(id='d', ctx=Load()), Name(id='e', ctx=Load())])) As one can see, it's the boolean operator or applied to four sub-expressions: comparison a == b; and simple expressions c, d, and e.


Instruction:
NameError: name 'self' is not defined

Why such structure class A: def __init__(self, a): self.a = a def p(self, b=self.a): print b gives an error NameError: name 'self' is not defined?

Output: Default argument values are evaluated at function define-time, but self is an argument only available at function call time. Thus arguments in the argument list cannot refer each other. It's a common pattern to default an argument to None and add a test for that in code: def p(self, b=None): if b is None: b = self.a print b Update 2022: Python developers are now considering late-bound argument defaults for future Python versions.


Instruction:
How to overwrite the previous print to stdout?

If I had the following code: for x in range(10): print(x) I would get the output of 1 2 etc.. What I would like to do is instead of printing a newline, I want to replace the previous value and overwrite it with the new value on the same line.

Output: Simple Version One way is to use the carriage return ('\r') character to return to the start of the line without advancing to the next line. Python 3 for x in range(10): print(x, end='\r') print() Python 2.7 forward compatible from __future__ import print_function for x in range(10): print(x, end='\r') print() Python 2.7 for x in range(10): print '{}\r'.format(x), print Python 2.0-2.6 for x in range(10): print '{0}\r'.format(x), print In the latter two (Python 2-only) cases, the comma at the end of the print statement tells it not to go to the next line. The last print statement advances to the next line so your prompt won't overwrite your final output. Line Cleaning If you can’t guarantee that the new line of text is not shorter than the existing line, then you just need to add a “clear to end of line” escape sequence, '\x1b[1K' ('\x1b' = ESC): for x in range(75): print('*' * (75 - x), x, end='\x1b[1K\r') print() Long Line Wrap All these methods assume you’re not writing more than the length of the line. The carriage return only returns to the start of the current line, so if your output is longer than a line, you’ll only erase the last line. If this is enough of a problem that you need to control it, you can disable line wrapping to keep the cursor from wrapping to the next line. (Instead, the cursor sticks to the end of the line, and successive characters overwrite.) Line wrap is disabled with print('\x1b[7l', end='') and re-enabled with print('\x1b[7h', end=''). Note that there is no automatic re-enable of line wrap at any point: don’t leave the terminal broken if an exception ends your program!


Instruction:
How do I avoid HTTP error 403 when web scraping with Python?

When I try this code to scrape a web page: #import requests import urllib.request from bs4 import BeautifulSoup #from urllib import urlopen import re webpage = urllib.request.urlopen('http://www.cmegroup.com/trading/products/#sortField=oi&sortAsc=false&venues=3&page=1&cleared=1&group=1').read findrows = re.compile('
(.*?)
') findlink = re.compile('(.*)') row_array = re.findall(findrows, webpage) links = re.finall(findlink, webpate) print(len(row_array)) iterator = [] I get an error like: File "C:\Python33\lib\urllib\request.py", line 160, in urlopen return opener.open(url, data, timeout) File "C:\Python33\lib\urllib\request.py", line 479, in open response = meth(req, response) File "C:\Python33\lib\urllib\request.py", line 591, in http_response 'http', request, response, code, msg, hdrs) File "C:\Python33\lib\urllib\request.py", line 517, in error return self._call_chain(*args) File "C:\Python33\lib\urllib\request.py", line 451, in _call_chain result = func(*args) File "C:\Python33\lib\urllib\request.py", line 599, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 403: Forbidden Does the website think I'm a bot? How can I fix the problem?

Output: This is probably because of mod_security or some similar server security feature which blocks known spider/bot user agents (urllib uses something like python urllib/3.3.0, it's easily detected). Try setting a known browser user agent with: from urllib.request import Request, urlopen req = Request( url='http://www.cmegroup.com/trading/products/#sortField=oi&sortAsc=false&venues=3&page=1&cleared=1&group=1', headers={'User-Agent': 'Mozilla/5.0'} ) webpage = urlopen(req).read() This works for me. By the way, in your code you are missing the () after .read in the urlopen line, but I think that it's a typo. TIP: since this is exercise, choose a different, non restrictive site. Maybe they are blocking urllib for some reason...


Instruction:
How to calculate the MD5 checksum of a file in Python?

I have written some code in Python that checks for an MD5 hash in a file and makes sure the hash matches that of the original. Here is what I have developed: # Defines filename filename = "file.exe" # Gets MD5 from file def getmd5(filename): return m.hexdigest() md5 = dict() for fname in filename: md5[fname] = getmd5(fname) # If statement for alerting the user whether the checksum passed or failed if md5 == '>md5 will go here<': print("MD5 Checksum passed. You may now close this window") input ("press enter") else: print("MD5 Checksum failed. Incorrect MD5 in file 'filename'. Please download a new copy") input("press enter") exit But whenever I run the code, I get the following error: Traceback (most recent call last): File "C:\Users\Username\md5check.py", line 13, in <module> md5[fname] = getmd5(fname) File "C:\Users\Username\md5check.py, line 9, in getmd5 return m.hexdigest() NameError: global name 'm' is not defined Is there anything I am missing in my code?

Output: In regards to your error and what's missing in your code. m is a name which is not defined for getmd5() function. No offence, I know you are a beginner, but your code is all over the place. Let's look at your issues one by one :) First, you are not using hashlib.md5.hexdigest() method correctly. Please refer explanation on hashlib functions in Python Doc Library. The correct way to return MD5 for provided string is to do something like this: >>> import hashlib >>> hashlib.md5("example string").hexdigest() '2a53375ff139d9837e93a38a279d63e5' However, you have a bigger problem here. You are calculating MD5 on a file name string, where in reality MD5 is calculated based on file contents. You will need to basically read file contents and pipe it though MD5. My next example is not very efficient, but something like this: >>> import hashlib >>> hashlib.md5(open('filename.exe','rb').read()).hexdigest() 'd41d8cd98f00b204e9800998ecf8427e' As you can clearly see second MD5 hash is totally different from the first one. The reason for that is that we are pushing contents of the file through, not just file name. A simple solution could be something like that: # Import hashlib library (md5 method is part of it) import hashlib # File to check file_name = 'filename.exe' # Correct original md5 goes here original_md5 = '5d41402abc4b2a76b9719d911017c592' # Open,close, read file and calculate MD5 on its contents with open(file_name, 'rb') as file_to_check: # read contents of the file data = file_to_check.read() # pipe contents of the file through md5_returned = hashlib.md5(data).hexdigest() # Finally compare original MD5 with freshly calculated if original_md5 == md5_returned: print "MD5 verified." else: print "MD5 verification failed!." Please look at the post Python: Generating a MD5 checksum of a file. It explains in detail a couple of ways how it can be achieved efficiently. Best of luck.


Instruction:
How to get a value from a Pandas DataFrame and not the index and object type

Say I have the following DataFrame Letter Number A 1 B 2 C 3 D 4 Which can be obtained through the following code import pandas as pd letters = pd.Series(('A', 'B', 'C', 'D')) numbers = pd.Series((1, 2, 3, 4)) keys = ('Letters', 'Numbers') df = pd.concat((letters, numbers), axis=1, keys=keys) Now I want to get the value C from the column Letters. The command line df[df.Letters=='C'].Letters will return 2 C Name: Letters, dtype: object How can I get only the value C and not the whole two line output?

Output: df[df.Letters=='C'].Letters.item() This returns the first element in the Index/Series returned from that selection. In this case, the value is always the first element. EDIT: Or you can run a loc() and access the first element that way. This was shorter and is the way I have implemented it in the past. Pandas Index doc Pandas Series doc


Instruction:
Use .corr to get the correlation between two columns

I have the following pandas dataframe Top15: I create a column that estimates the number of citable documents per person: Top15['PopEst'] = Top15['Energy Supply'] / Top15['Energy Supply per Capita'] Top15['Citable docs per Capita'] = Top15['Citable documents'] / Top15['PopEst'] I want to know the correlation between the number of citable documents per capita and the energy supply per capita. So I use the .corr() method (Pearson's correlation): data = Top15[['Citable docs per Capita','Energy Supply per Capita']] correlation = data.corr(method='pearson') I want to return a single number, but the result is:

Output: Without actual data it is hard to answer the question but I guess you are looking for something like this: Top15['Citable docs per Capita'].corr(Top15['Energy Supply per Capita']) That calculates the correlation between your two columns 'Citable docs per Capita' and 'Energy Supply per Capita'. To give an example: import pandas as pd df = pd.DataFrame({'A': range(4), 'B': [2*i for i in range(4)]}) A B 0 0 0 1 1 2 2 2 4 3 3 6 Then df['A'].corr(df['B']) gives 1 as expected. Now, if you change a value, e.g. df.loc[2, 'B'] = 4.5 A B 0 0 0.0 1 1 2.0 2 2 4.5 3 3 6.0 the command df['A'].corr(df['B']) returns 0.99586 which is still close to 1, as expected. If you apply .corr() directly to your dataframe, it will return all pairwise correlations between your columns; that's why you then observe 1s at the diagonal of your matrix (each column is perfectly correlated with itself). df.corr() will therefore return A B A 1.000000 0.995862 B 0.995862 1.000000 In the graphic you show, only the upper left corner of the correlation matrix is represented (I assume). There can be cases, where you get NaNs in your solution - check this post for an example. If you want to filter entries above/below a certain threshold, you can check this question. If you want to plot a heatmap of the correlation coefficients, you can check this answer and if you then run into the issue with overlapping axis-labels check the following post.


Instruction:
How can I check if string input is a number?

How do I check if a user's string input is a number (e.g., -1, 0, 1, etc.)? user_input = input("Enter something:") if type(user_input) == int: print("Is a number") else: print("Not a number") The above won't work since input always returns a string.

Output: Simply try converting it to an int and then bailing out if it doesn't work. try: val = int(userInput) except ValueError: print("That's not an int!") See Handling Exceptions in the official tutorial.


Instruction:
In Python, how do I read the exif data for an image?

I'm using PIL. How do I turn the EXIF data of a picture into a dictionary?

Output: You can use the _getexif() protected method of a PIL Image. import PIL.Image img = PIL.Image.open('img.jpg') exif_data = img._getexif() This should give you a dictionary indexed by EXIF numeric tags. If you want the dictionary indexed by the actual EXIF tag name strings, try something like: import PIL.ExifTags exif = { PIL.ExifTags.TAGS[k]: v for k, v in img._getexif().items() if k in PIL.ExifTags.TAGS }


Instruction:
Join a list of strings in python and wrap each string in quotation marks

I've got: words = ['hello', 'world', 'you', 'look', 'nice'] I want to have: '"hello", "world", "you", "look", "nice"' What's the easiest way to do this with Python?

Output: Update 2021: With f strings in Python3 >>> words = ['hello', 'world', 'you', 'look', 'nice'] >>> ', '.join(f'"{w}"' for w in words) '"hello", "world", "you", "look", "nice"' Original Answer (Supports Python 2.6+) >>> words = ['hello', 'world', 'you', 'look', 'nice'] >>> ', '.join('"{0}"'.format(w) for w in words) '"hello", "world", "you", "look", "nice"'


Instruction:
Bypass confirmation prompt for pip uninstall

I'm trying to uninstall all django packages in my superuser environment to ensure that all my webapp dependencies are installed to my virtualenv. sudo su sudo pip freeze | grep -E '^django-' | xargs pip -q uninstall But pip wants to confirm every package uninstall, and there doesn't seem to be a -y option for pip. Is there a better way to uninstall a batch of python modules? Is rm -rf .../site-packages/ a proper way to go? Is there an easy_install alternative? Alternatively, would it be better to force pip to install all dependencies to the virtualenv rather than relying on the system python modules to meet those dependencies, e.g. pip --upgrade install, but forcing even equally old versions to be installed to override any system modules. I tried activating my virtualenv and then pip install --upgrade -r requirements.txt and that does seem to install the dependencies, even those existing in my system path, but I can't be sure if that's because my system modules were old. And man pip doesn't seem to guarantee this behavior (i.e. installing the same version of a package that already exists in the system site-packages).

Output: starting with pip version 7.1.2 you can run pip uninstall -y <python package(s)> pip uninstall -y package1 package2 package3 or from file pip uninstall -y -r requirements.txt


Instruction:
How to find children of nodes using BeautifulSoup

I want to get all the  tags which are children of
:

 link1

 link2

 I know how to find element with particular class like this: soup.find("li", { "class" : "test" }) But I don't know how to find all  which are children of
 but not any others. Like I want to select: link1

Output: Try this li = soup.find('li', {'class': 'test'}) children = li.findChildren("a" , recursive=False) for child in children: print(child)


Instruction:
Get default line color cycle

I noticed when you plot that the first line is blue, then orange, then green, and so on. Is there some way to access this list of colors? I've seen a million posts on how to change the color cycle or access the iterator, but not on how to just get the list of colors that matplotlib cycles through by default.

Output: In matplotlib versions >= 1.5, you can print the rcParam called axes.prop_cycle: print(plt.rcParams['axes.prop_cycle'].by_key()['color']) # [u'#1f77b4', u'#ff7f0e', u'#2ca02c', u'#d62728', u'#9467bd', u'#8c564b', u'#e377c2', u'#7f7f7f', u'#bcbd22', u'#17becf'] Or equivalently, in python2: print plt.rcParams['axes.prop_cycle'].by_key()['color'] In versions < 1.5, this was called color_cycle: print plt.rcParams['axes.color_cycle'] # [u'b', u'g', u'r', u'c', u'm', u'y', u'k'] Note that the default color cycle changed in version 2.0.0 http://matplotlib.org/users/dflt_style_changes.html#colors-in-default-property-cycle


Instruction:
FastAPI throws an error (Error loading ASGI app. Could not import module "api")

I tried to run FastAPI using uvicorn webserver but it throws an error. I run this command, uvicorn api:app --reload --host 0.0.0.0 but there is an error in the terminal. Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit) Started reloader process [23445] Error loading ASGI app. Could not import module "api". Stopping reloader process [23445]

Output: TL;DR Add the directory name in front of your filename uvicorn src.main:app or cd into that directory cd src uvicorn main:app Long Answer It happens because you are not in the same folder with your FastAPI app instance more specifically: Let's say i have an app-tree like this; my_fastapi_app/ ├── app.yaml ├── docker-compose.yml ├── src │ └── main.py └── tests ├── test_xx.py └── test_yy.py $ pwd # Present Working Directory /home/yagiz/Desktop/my_fastapi_app I'm not inside the same folder with my app instance, so if I try to run my app with uvicorn I'll get an error like yours $ uvicorn main:app --reload INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit) INFO: Started reloader process [40645] using statreload ERROR: Error loading ASGI app. Could not import module "main". The answer is so simple, add the folder name in front of your filename uvicorn src.main:app --reload or you can change your working directory cd src Now i'm inside of the folder with my app instance src └── main.py Run your uvicorn again $ uvicorn main:app --reload INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit) INFO: Started reloader process [40726] using statreload INFO: Started server process [40728] INFO: Waiting for application startup. INFO: Application startup complete.


Instruction:
Example use of "continue" statement in Python?

The definition of the continue statement is: The continue statement continues with the next iteration of the loop. I can't find any good examples of code. Could someone suggest some simple cases where continue is necessary?

Output: Here's a simple example: for letter in 'Django': if letter == 'D': continue print("Current Letter: " + letter) Output will be: Current Letter: j Current Letter: a Current Letter: n Current Letter: g Current Letter: o It skips the rest of the current iteration (here: print) and continues to the next iteration of the loop.


Instruction:
pip freeze creates some weird path instead of the package version

I am working on developing a python package. I use pip freeze > requirements.txt to add the required package into the requirement.txt file. However, I realized that some of the packages, instead of the package version, have some path in front of them. numpy==1.19.0 packaging==20.4 pandas @ file:///opt/concourse/worker/volumes/live/38d1301c-8fa9-4d2f-662e-34dddf33b183/volume/pandas_1592841668171/work pandocfilters==1.4.2 Whereas, inside the environment, I get: >>> pandas.__version__ '1.0.5' Do you have any idea how to address this problem?

Output: It looks like this is an open issue with pip freeze in version 20.1, the current workaround is to use: pip list --format=freeze > requirements.txt In a nutshell, this is caused by changing the behavior of pip freeze to include direct references for distributions installed from direct URL references. You can read more about the issue on GitHub: pip freeze does not show version for in-place installs Output of "pip freeze" and "pip list --format=freeze" differ for packages installed via Direct URLs Better freeze of distributions installed from direct URL references


Instruction:
Retrieving subfolders names in S3 bucket from Boto3

Using Boto3, I can access my AWS S3 bucket: s3 = boto3.resource('s3') bucket = s3.Bucket('my-bucket-name') Now, the bucket contains folder first-level, which itself contains several sub-folders named with a timestamp, for instance 1456753904534. I need to know the name of these sub-folders for another job I'm doing and I wonder whether I could have boto3 retrieve those for me. So I tried: objs = bucket.meta.client.list_objects(Bucket='my-bucket-name') which gives a dictionary, whose key 'Contents' gives me all the third-level files instead of the second-level timestamp directories, in fact I get a list containing things as {u'ETag': '"etag"', u'Key': first-level/1456753904534/part-00014', u'LastModified': datetime.datetime(2016, 2, 29, 13, 52, 24, tzinfo=tzutc()), u'Owner': {u'DisplayName': 'owner', u'ID': 'id'}, u'Size': size, u'StorageClass': 'storageclass'} You can see that the specific files, in this case part-00014 are retrieved, while I'd like to get the name of the directory alone. In principle I could strip out the directory name from all the paths but it's ugly and expensive to retrieve everything at third level to get the second level! I also tried something reported here: for o in bucket.objects.filter(Delimiter='/'): print(o.key) but I do not get the folders at the desired level. Is there a way to solve this?

Output: S3 is an object storage, it doesn't have real directory structure. The "/" is rather cosmetic. One reason that people want to have a directory structure, because they can maintain/prune/add a tree to the application. For S3, you treat such structure as sort of index or search tag. To manipulate object in S3, you need boto3.client or boto3.resource, e.g. To list all object import boto3 s3 = boto3.client("s3") all_objects = s3.list_objects(Bucket = 'bucket-name') http://boto3.readthedocs.org/en/latest/reference/services/s3.html#S3.Client.list_objects In fact, if the s3 object name is stored using '/' separator. The more recent version of list_objects (list_objects_v2) allows you to limit the response to keys that begin with the specified prefix. To limit the items to items under certain sub-folders: import boto3 s3 = boto3.client("s3") response = s3.list_objects_v2( Bucket=BUCKET, Prefix ='DIR1/DIR2', MaxKeys=100 ) Documentation Another option is using python os.path function to extract the folder prefix. Problem is that this will require listing objects from undesired directories. import os s3_key = 'first-level/1456753904534/part-00014' filename = os.path.basename(s3_key) foldername = os.path.dirname(s3_key) # if you are not using conventional delimiter like '#' s3_key = 'first-level#1456753904534#part-00014' filename = s3_key.split("#")[-1] A reminder about boto3 : boto3.resource is a nice high level API. There are pros and cons using boto3.client vs boto3.resource. If you develop internal shared library, using boto3.resource will give you a blackbox layer over the resources used.


Instruction:
How do you check in Python whether a string contains only numbers?

How do you check whether a string contains only numbers? I've given it a go here. I'd like to see the simplest way to accomplish this. import string def main(): isbn = input("Enter your 10 digit ISBN number: ") if len(isbn) == 10 and string.digits == True: print ("Works") else: print("Error, 10 digit number was not inputted and/or letters were inputted.") main() if __name__ == "__main__": main() input("Press Enter to exit: ")

Output: You'll want to use the isdigit method on your str object: if len(isbn) == 10 and isbn.isdigit(): From the isdigit documentation: str.isdigit() Return True if all characters in the string are digits and there is at least one character, False otherwise. Digits include decimal characters and digits that need special handling, such as the compatibility superscript digits. This covers digits which cannot be used to form numbers in base 10, like the Kharosthi numbers. Formally, a digit is a character that has the property value Numeric_Type=Digit or Numeric_Type=Decimal.


Instruction:
Insert an element at a specific index in a list and return the updated list

I have this: >>> a = [1, 2, 4] >>> print(a) [1, 2, 4] >>> print(a.insert(2, 3)) None >>> print(a) [1, 2, 3, 4] >>> b = a.insert(3, 6) >>> print(b) None >>> print a [1, 2, 3, 6, 4] Is there a way I can get the updated list as the result, instead of updating the original list in place?

Output: The shortest I got: b = a[:2] + [3] + a[2:] >>> >>> a = [1, 2, 4] >>> print a [1, 2, 4] >>> b = a[:2] + [3] + a[2:] >>> print a [1, 2, 4] >>> print b [1, 2, 3, 4]


Instruction:
How can I profile Python code line-by-line?

I've been using cProfile to profile my code, and it's been working great. I also use gprof2dot.py to visualize the results (makes it a little clearer). However, cProfile (and most other Python profilers I've seen so far) seem to only profile at the function-call level. This causes confusion when certain functions are called from different places - I have no idea if call #1 or call #2 is taking up the majority of the time. This gets even worse when the function in question is six levels deep, called from seven other places. How do I get a line-by-line profiling? Instead of this: function #12, total time: 2.0s I'd like to see something like this: function #12 (called from somefile.py:102) 0.5s function #12 (called from main.py:12) 1.5s cProfile does show how much of the total time "transfers" to the parent, but again this connection is lost when you have a bunch of layers and interconnected calls. Ideally, I'd love to have a GUI that would parse through the data, then show me my source file with a total time given to each line. Something like this: main.py: a = 1 # 0.0s result = func(a) # 0.4s c = 1000 # 0.0s result = func(c) # 5.0s Then I'd be able to click on the second "func(c)" call to see what's taking up time in that call, separate from the "func(a)" call. Does that make sense?

Output: I believe that's what Robert Kern's line_profiler is intended for. From the link: File: pystone.py Function: Proc2 at line 149 Total time: 0.606656 s Line # Hits Time Per Hit % Time Line Contents ============================================================== 149 @profile 150 def Proc2(IntParIO): 151 50000 82003 1.6 13.5 IntLoc = IntParIO + 10 152 50000 63162 1.3 10.4 while 1: 153 50000 69065 1.4 11.4 if Char1Glob == 'A': 154 50000 66354 1.3 10.9 IntLoc = IntLoc - 1 155 50000 67263 1.3 11.1 IntParIO = IntLoc - IntGlob 156 50000 65494 1.3 10.8 EnumLoc = Ident1 157 50000 68001 1.4 11.2 if EnumLoc == Ident1: 158 50000 63739 1.3 10.5 break 159 50000 61575 1.2 10.1 return IntParIO


Instruction:
How to enable a virtualenv in a systemd service unit?

I want to "activate" a virtualenv in a systemd service file. I would like to avoid having a shell process between the systemd process and the python interpreter. My current solution looks like this: [Unit] Description=fooservice After=syslog.target network.target [Service] Type=simple User=fooservice WorkingDirectory={{ venv_home }} ExecStart={{ venv_home }}/fooservice --serve-in-foreground Restart=on-abort EnvironmentFile=/etc/sysconfig/fooservice.env [Install] WantedBy=multi-user.target /etc/sysconfig/fooservice.env PATH={{ venv_home }}/bin:/usr/local/bin:/usr/bin:/bin PYTHONIOENCODING=utf-8 PYTHONPATH={{ venv_home }}/... VIRTUAL_ENV={{ venv_home }} But I am having trouble. I get ImportErrors since some entries in sys.path are missing.

Output: The virtualenv is "baked into the Python interpreter in the virtualenv". So for most cases this means you can launch python or console_scripts directly in that virtualenv and don't need to activate the virtualenv first or manage PATH yourself. Note: Replace {{ venv_home }} with the venv home dir to use: ExecStart={{ venv_home }}/bin/fooservice --serve-in-foreground or ExecStart={{ venv_home }}/bin/python {{ venv_home }}/fooservice.py --serve-in-foreground and remove the EnvironmentFile entry. To verify that it is indeed correct you can check sys.path by running {{ venv_home }}/bin/python -m site and comparing the output to python -m site


Instruction:
Print a string as hexadecimal bytes

I have this string: Hello, World! and I want to print it using Python as '48:65:6c:6c:6f:2c:20:57:6f:72:6c:64:21'. hex() works only for integers. How can it be done?

Output: You can transform your string to an integer generator. Apply hexadecimal formatting for each element and intercalate with a separator: >>> s = "Hello, World!" >>> ":".join("{:02x}".format(ord(c)) for c in s) '48:65:6c:6c:6f:2c:20:57:6f:72:6c:64:21


Instruction:
What is the SQL ''LIKE" equivalent on Django ORM queries?

What is the equivalent of the following SQL statement in Django? SELECT * FROM table_name WHERE string LIKE pattern; I tried this: result = table.objects.filter( pattern in string ) but it didn't work. How can I implement it?

Output: Use __contains or __icontains (case-insensitive): result = table.objects.filter(string__contains='pattern') The SQL equivalent is SELECT ... WHERE string LIKE '%pattern%'; @Dmitri's answer below covers patterns like 'pattern%' or '%pattern'


Instruction:
How to rotate x-axis tick labels in a pandas plot

With the following code: import matplotlib matplotlib.style.use('ggplot') import matplotlib.pyplot as plt import pandas as pd df = pd.DataFrame({ 'celltype':["foo","bar","qux","woz"], 's1':[5,9,1,7], 's2':[12,90,13,87]}) df = df[["celltype","s1","s2"]] df.set_index(["celltype"],inplace=True) df.plot(kind='bar',alpha=0.75) plt.xlabel("") I made this plot: How can I rotate the x-axis tick labels to 0 degrees? I tried adding this but did not work: plt.set_xticklabels(df.index,rotation=90)

Output: Pass param rot=0 to rotate the xticklabels: import matplotlib matplotlib.style.use('ggplot') import matplotlib.pyplot as plt import pandas as pd df = pd.DataFrame({ 'celltype':["foo","bar","qux","woz"], 's1':[5,9,1,7], 's2':[12,90,13,87]}) df = df[["celltype","s1","s2"]] df.set_index(["celltype"],inplace=True) df.plot(kind='bar',alpha=0.75, rot=0) plt.xlabel("") plt.show() yields plot:


Instruction:
Apply function to each element of a list

Suppose I have a list like: mylis = ['this is test', 'another test'] How do I apply a function to each element in the list? For example, how do I apply str.upper to get: ['THIS IS TEST', 'ANOTHER TEST']

Output: Using the built-in standard library map: >>> mylis = ['this is test', 'another test'] >>> list(map(str.upper, mylis)) ['THIS IS TEST', 'ANOTHER TEST'] In Python 2.x, map constructed the desired new list by applying a given function to every element in a list. In Python 3.x, map constructs an iterator instead of a list, so the call to list is necessary. If you are using Python 3.x and require a list the list comprehension approach would be better suited.


Instruction:
Flask ImportError: No Module Named Flask

I'm following the Flask tutorial here: http://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-i-hello-world I get to the point where I try ./run.py and I get: Traceback (most recent call last): File "./run.py", line 3, in <module> from app import app File "/Users/benjaminclayman/Desktop/microblog/app/__init__.py", line 1, in <module> from flask import Flask ImportError: No module named flask This looks similar to: ImportError: No module named flask But their solutions aren't helpful. For reference, I do have a folder named flask which one user mentioned may cause issues.

Output: Try deleting the virtualenv you created. Then create a new virtualenv with: virtualenv flask Then: cd flask Now let's activate the virtualenv source bin/activate Now you should see (flask) on the left of the command line. Edit: In windows there is no "source" that's a linux thing, instead execute the activate.bat file, here I do it using Powershell: PS C:\DEV\aProject> & .\Flask\Scripts\activate) Let's install flask: pip install flask Then create a file named hello.py (NOTE: see UPDATE Flask 1.0.2 below): from flask import Flask app = Flask(__name__) @app.route("/") def hello(): return "Hello World!" if __name__ == "__main__": app.run() and run it with: python hello.py UPDATE Flask 1.0.2 With the new flask release there is no need to run the app from your script. hello.py should look like this now: from flask import Flask app = Flask(__name__) @app.route("/") def hello(): return "Hello World!" and run it with: FLASK_APP=hello.py flask run Make sure to be inside the folder where hello.py is when running the latest command. All the steps before the creation of the hello.py apply for this case as well


Instruction:
How to filter objects for count annotation in Django?

Consider simple Django models Event and Participant: class Event(models.Model): title = models.CharField(max_length=100) class Participant(models.Model): event = models.ForeignKey(Event, db_index=True) is_paid = models.BooleanField(default=False, db_index=True) It's easy to annotate events query with total number of participants: events = Event.objects.all().annotate(participants=models.Count('participant')) How to annotate with count of participants filtered by is_paid=True? I need to query all events regardless of number of participants, e.g. I don't need to filter by annotated result. If there are 0 participants, that's ok, I just need 0 in annotated value. The example from documentation doesn't work here, because it excludes objects from query instead of annotating them with 0. Update. Django 1.8 has new conditional expressions feature, so now we can do like this: events = Event.objects.all().annotate(paid_participants=models.Sum( models.Case( models.When(participant__is_paid=True, then=1), default=0, output_field=models.IntegerField() ))) Update 2. Django 2.0 has new Conditional aggregation feature, see the accepted answer below. This also works in Django 3.x

Output: Conditional aggregation in Django 2.0+ allows you to further reduce the amount of faff this has been in the past. This will also use Postgres' filter logic, which is somewhat faster than a sum-case (I've seen numbers like 20-30% bandied around). Anyway, in your case, we're looking at something as simple as: from django.db.models import Q, Count events = Event.objects.annotate( paid_participants=Count('participant', filter=Q(participant__is_paid=True)) ) There's a separate section in the docs about filtering on annotations. It's the same stuff as conditional aggregation but more like my example above. Either which way, this is a lot healthier than the gnarly subqueries I was doing before. For more complex annotation filters, it may be more understandable to structure the filter as a separate queryset that is passed to an __in expression: from django.db.models import Q, Count paid_participants = Participant.objects.filter(is_paid=True) events = Event.objects.annotate( paid_participants=Count( 'participant', filter=Q(participant__in=paid_participants) ) ) In this context, the Participant queryset does not need to explicitly filter on the outer Event ID like a subquery because Count (and other aggregation functions) implicitly filter for reverse foreign key relationships of the current row. In other words, with this format specify your filters globally for the Model that is being counted.


Instruction:
Given a URL to a text file, what is the simplest way to read the contents of the text file?

In Python, when given the URL for a text file, what is the simplest way to access the contents off the text file and print the contents of the file out locally line-by-line without saving a local copy of the text file? TargetURL=http://www.myhost.com/SomeFile.txt #read the file #print first line #print second line #etc

Output: Edit 09/2016: In Python 3 and up use urllib.request instead of urllib2 Actually the simplest way is: import urllib2 # the lib that handles the url stuff data = urllib2.urlopen(target_url) # it's a file like object and works just like a file for line in data: # files are iterable print line You don't even need "readlines", as Will suggested. You could even shorten it to: * import urllib2 for line in urllib2.urlopen(target_url): print line But remember in Python, readability matters. However, this is the simplest way but not the safe way because most of the time with network programming, you don't know if the amount of data to expect will be respected. So you'd generally better read a fixed and reasonable amount of data, something you know to be enough for the data you expect but will prevent your script from been flooded: import urllib2 data = urllib2.urlopen("http://www.google.com").read(20000) # read only 20 000 chars data = data.split("\n") # then split it into lines for line in data: print line * Second example in Python 3: import urllib.request # the lib that handles the url stuff for line in urllib.request.urlopen(target_url): print(line.decode('utf-8')) #utf-8 or iso8859-1 or whatever the page encoding scheme is


Instruction:
Display all dataframe columns in a Jupyter Python Notebook

I want to show all columns in a dataframe in a Jupyter Notebook. Jupyter shows some of the columns and adds dots to the last columns like in the following picture: How can I display all columns?

Output: Try the display max_columns setting as follows: import pandas as pd from IPython.display import display df = pd.read_csv("some_data.csv") pd.options.display.max_columns = None display(df) Or pd.set_option('display.max_columns', None) Pandas 0.11.0 backwards This is deprecated but in versions of Pandas older than 0.11.0 the max_columns setting is specified as follows: pd.set_printoptions(max_columns=500)


Instruction:
How can I make a python dataclass hashable?

I have a dataclass whose instances I want to hash and order, using the id member as a key. from dataclasses import dataclass, field @dataclass(eq=True, order=True) class Category: id: str = field(compare=True) name: str = field(default="set this in post_init", compare=False) I know that I can implement __hash__ myself. However, I would like dataclasses to do the work for me because they are intended to handle this. Unfortunately, the above dataclass fails: a = sorted(list(set([ Category(id='x'), Category(id='y')]))) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: unhashable type: 'Category'

Output: From the docs: Here are the rules governing implicit creation of a __hash__() method: [...] If eq and frozen are both true, by default dataclass() will generate a __hash__() method for you. If eq is true and frozen is false, __hash__() will be set to None, marking it unhashable (which it is, since it is mutable). If eq is false, __hash__() will be left untouched meaning the __hash__() method of the superclass will be used (if the superclass is object, this means it will fall back to id-based hashing). Since you set eq=True and left frozen at the default (False), your dataclass is unhashable. You have 3 options: Set frozen=True (in combination with the default eq=True), which will make your class immutable and hashable. @dataclass(frozen=True) Set unsafe_hash=True, which will create a __hash__ method but leave your class mutable. @dataclass(unsafe_hash=True) Mutability risks problems if an instance of your class is modified while stored in a dict or set: cat = Category('foo', 'bar') categories = {cat} cat.id = 'baz' print(cat in categories) # False Manually implement a __hash__ method.


Instruction:
'and' (boolean) vs '&' (bitwise) - Why difference in behavior with lists vs numpy arrays?

What explains the difference in behavior of boolean and bitwise operations on lists vs NumPy arrays? I'm confused about the appropriate use of & vs and in Python, illustrated in the following examples. mylist1 = [True, True, True, False, True] mylist2 = [False, True, False, True, False] >>> len(mylist1) == len(mylist2) True # ---- Example 1 ---- >>> mylist1 and mylist2 [False, True, False, True, False] # I would have expected [False, True, False, False, False] # ---- Example 2 ---- >>> mylist1 & mylist2 TypeError: unsupported operand type(s) for &: 'list' and 'list' # Why not just like example 1? >>> import numpy as np # ---- Example 3 ---- >>> np.array(mylist1) and np.array(mylist2) ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() # Why not just like Example 4? # ---- Example 4 ---- >>> np.array(mylist1) & np.array(mylist2) array([False, True, False, False, False], dtype=bool) # This is the output I was expecting! This answer and this answer helped me understand that and is a boolean operation but & is a bitwise operation. I read about bitwise operations to better understand the concept, but I am struggling to use that information to make sense of my above 4 examples. Example 4 led me to my desired output, so that is fine, but I am still confused about when/how/why I should use and vs &. Why do lists and NumPy arrays behave differently with these operators? Can anyone help me understand the difference between boolean and bitwise operations to explain why they handle lists and NumPy arrays differently?

Output: and tests whether both expressions are logically True while & (when used with True/False values) tests if both are True. In Python, empty built-in objects are typically treated as logically False while non-empty built-ins are logically True. This facilitates the common use case where you want to do something if a list is empty and something else if the list is not. Note that this means that the list [False] is logically True: >>> if [False]: ... print('True') ... True So in Example 1, the first list is non-empty and therefore logically True, so the truth value of the and is the same as that of the second list. (In our case, the second list is non-empty and therefore logically True, but identifying that would require an unnecessary step of calculation.) For example 2, lists cannot meaningfully be combined in a bitwise fashion because they can contain arbitrary unlike elements. Things that can be combined bitwise include: Trues and Falses, integers. NumPy objects, by contrast, support vectorized calculations. That is, they let you perform the same operations on multiple pieces of data. Example 3 fails because NumPy arrays (of length > 1) have no truth value as this prevents vector-based logic confusion. Example 4 is simply a vectorized bit and operation. Bottom Line If you are not dealing with arrays and are not performing math manipulations of integers, you probably want and. If you have vectors of truth values that you wish to combine, use numpy with &.


Instruction:
Resolving new pip backtracking runtime issue

The new pip dependency resolver that was released with version 20.3 takes an inappropriately long time to install a package. On our CI pipeline yesterday, a docker build that used to take ~10 minutes timed out after 1h of pip installation messages like this (almost for every library that is installed by any dependency there is a similar log output): INFO: pip is looking at multiple versions of setuptools to determine which version is compatible with other requirements. This could take a while. Downloading setuptools-50.0.0-py3-none-any.whl (783 kB) Downloading setuptools-49.6.0-py3-none-any.whl (803 kB) Downloading setuptools-49.5.0-py3-none-any.whl (803 kB) Downloading setuptools-49.4.0-py3-none-any.whl (803 kB) Downloading setuptools-49.3.2-py3-none-any.whl (790 kB) INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking Downloading setuptools-49.3.1-py3-none-any.whl (790 kB) Downloading setuptools-49.3.0-py3-none-any.whl (790 kB) Downloading setuptools-49.2.1-py3-none-any.whl (789 kB) Downloading setuptools-49.2.0-py3-none-any.whl (789 kB) Downloading setuptools-49.1.3-py3-none-any.whl (789 kB) Downloading setuptools-49.1.2-py3-none-any.whl (789 kB) Downloading setuptools-49.1.1-py3-none-any.whl (789 kB) Downloading setuptools-49.1.0-py3-none-any.whl (789 kB) Downloading setuptools-49.0.1-py3-none-any.whl (789 kB) Downloading setuptools-49.0.0-py3-none-any.whl (789 kB) Downloading setuptools-48.0.0-py3-none-any.whl (786 kB) Downloading setuptools-47.3.2-py3-none-any.whl (582 kB) Downloading setuptools-47.3.1-py3-none-any.whl (582 kB) Downloading setuptools-47.3.0-py3-none-any.whl (583 kB) Downloading setuptools-47.2.0-py3-none-any.whl (583 kB) Downloading setuptools-47.1.1-py3-none-any.whl (583 kB) Downloading setuptools-47.1.0-py3-none-any.whl (583 kB) Downloading setuptools-47.0.0-py3-none-any.whl (583 kB) Downloading setuptools-46.4.0-py3-none-any.whl (583 kB) Downloading setuptools-46.3.1-py3-none-any.whl (582 kB) Downloading setuptools-46.3.0-py3-none-any.whl (582 kB) Downloading setuptools-46.2.0-py3-none-any.whl (582 kB) Downloading setuptools-46.1.3-py3-none-any.whl (582 kB) Downloading setuptools-46.1.2-py3-none-any.whl (582 kB) Downloading setuptools-46.1.1-py3-none-any.whl (582 kB) Downloading setuptools-46.1.0-py3-none-any.whl (582 kB) Downloading setuptools-46.0.0-py3-none-any.whl (582 kB) Downloading setuptools-45.3.0-py3-none-any.whl (585 kB) Downloading setuptools-45.2.0-py3-none-any.whl (584 kB) Downloading setuptools-45.1.0-py3-none-any.whl (583 kB) Downloading setuptools-45.0.0-py2.py3-none-any.whl (583 kB) Downloading setuptools-44.1.1-py2.py3-none-any.whl (583 kB) Downloading setuptools-44.1.0-py2.py3-none-any.whl (583 kB) Downloading setuptools-44.0.0-py2.py3-none-any.whl (583 kB) Downloading setuptools-43.0.0-py2.py3-none-any.whl (583 kB) Downloading setuptools-42.0.2-py2.py3-none-any.whl (583 kB) Downloading setuptools-42.0.1-py2.py3-none-any.whl (582 kB) Downloading setuptools-42.0.0-py2.py3-none-any.whl (582 kB) Downloading setuptools-41.6.0-py2.py3-none-any.whl (582 kB) Downloading setuptools-41.5.1-py2.py3-none-any.whl (581 kB) Downloading setuptools-41.5.0-py2.py3-none-any.whl (581 kB) Downloading setuptools-41.4.0-py2.py3-none-any.whl (580 kB) Downloading setuptools-41.3.0-py2.py3-none-any.whl (580 kB) Downloading setuptools-41.2.0-py2.py3-none-any.whl (576 kB) Downloading setuptools-41.1.0-py2.py3-none-any.whl (576 kB) Downloading setuptools-41.0.1-py2.py3-none-any.whl (575 kB) Downloading setuptools-41.0.0-py2.py3-none-any.whl (575 kB) Downloading setuptools-40.9.0-py2.py3-none-any.whl (575 kB) Downloading setuptools-40.8.0-py2.py3-none-any.whl (575 kB) Downloading setuptools-40.7.3-py2.py3-none-any.whl (574 kB) Downloading setuptools-40.7.2-py2.py3-none-any.whl (574 kB) Downloading setuptools-40.7.1-py2.py3-none-any.whl (574 kB) Downloading setuptools-40.7.0-py2.py3-none-any.whl (573 kB) Downloading setuptools-40.6.3-py2.py3-none-any.whl (573 kB) Downloading setuptools-40.6.2-py2.py3-none-any.whl (573 kB) Downloading setuptools-40.6.1-py2.py3-none-any.whl (573 kB) Downloading setuptools-40.6.0-py2.py3-none-any.whl (573 kB) Downloading setuptools-40.5.0-py2.py3-none-any.whl (569 kB) Downloading setuptools-40.4.3-py2.py3-none-any.whl (569 kB) Downloading setuptools-40.4.2-py2.py3-none-any.whl (569 kB) Downloading setuptools-40.4.1-py2.py3-none-any.whl (569 kB) Downloading setuptools-40.4.0-py2.py3-none-any.whl (568 kB) Downloading setuptools-40.3.0-py2.py3-none-any.whl (568 kB) I am quite confused whether we are using the new pip resolver correctly, especially since - Substantial improvements in new resolver for performance, output and error messages, avoiding infinite loops, and support for constraints files. The behavior seen is described as backtracking in the release notes. I understand why it is there. It specifies that I can use a constraint file (looks like a requirements.txt) that fixes the version of the dependencies to reduce the runtime using pip install -c constraints.txt setup.py. What is the best way to produce this constraints file? Currently, the best way I can think of is running pip install setup.py locally in a new virtual environment, then using pip freeze > constraints.txt. However, this still takes a lot of time for the local install (it's been stuck for about 10 minutes now). The notes do mention that This means the “work” is done once during development process, and so will save users this work during deployment. With the old dependency resolver, I was able to install this package in less than a minute locally. What is the recommended process here? Edit: I just found out that some of the dependencies are pointing directly to out internal gitlab server. If I instead install directly from our internal package registry, it works in a couple of minutes again.

Output: Latest update (2022-02) There seems to be major update in pip just few days old (version 22.0, release notes + relevant issue on github). I haven't tested it in more detail but it really seems to me that they optimized installation order calculation in complex case in such way that it resolves many issues we all encountered earlier. But I will need more time to check it. Anyway, the rest of this answer is still valid and smart requirements pinning suitable for particular project is a good practice imo. Since I encountered similar issue I agree this is quite annoying. Backtracking might be useful feature but you don't want to wait hours to complete with uncertain success. I found several option that might help: Use the old resolver (--use-deprecated=legacy-resolver) proposed in the answer by @Daniel Davee, but this is more like temporary solution than a proper one. Skip resolving dependencies with --no-deps option. I would not recommend this generally but in some cases you can have a working set of packages versions although there are some conflicts. Reduce the number of versions pip will try to backtrack and be more strict on package dependencies. This means instead of putting e.g. numpy in my requirements.txt, I could try numpy >= 1.18.0 or be even more strict with numpy == 1.18.0. The strictness might help a lot. Check the following sources: Fixing conflicts Github pip discussion Reducing backtracking I still do not have a proper answer that would always help but the best practice for requirements.txt seems to "pin" package versions. I found pip-tools that could help you manage this even with constrains.txt (but I am in an experimental phase so I can not tell you more). Update (2021-04) It seems author of the question was able to fix the issue (something with custom gitlab server) but I would like to extend this answer since it might be useful for others. After reading and trying I ended up with pinning all my package versions to a specific one. This really should be the correct way. Although everything can still work without it, there might be cases where if you don't pin your dependencies, your package manager will silently install a new version (when it's released) with possible bugs or incompatibility (this happens to me with dask last this year). There are several tools which might help you, I would recommend one of these approaches: Easiest one with pipreqs pipreqs is a library which generates pip requirements.txt file based on imports of any project you can start by pip install pipreqs and runnning just pipreqs in your project root (or eventually with --force flag if your requirements already exists) it will easily create requirements.txt with pinned versions based on imports in your project and versions taken from your environment then you can at any time create new environment based on this requirements.txt This is really simple tool (you even do not need to write your requirements.txt). It does not allow you to create something complex (might not be a good choice for bigger projects), last week I found one strange behavior (see this) but generally I'm happy with this tool as it usually works perfectly. Using pip-tools There are several other tools commonly used like pip-tools, Pipenv or Poetry. You can read more in Faster Docker builds with pipenv, poetry, or pip-tools or Python Application Dependency Management in 2018 (older but seems still valid to me). And it still seems to me that the best option (although it depends on your project/use case) is pip-tools. You can (this is one option, see more in docs): create requirements.in (the same format as requirements.txt, it's up to you whether you pin some package dependency or not) then you can use it by pip install pip-tools and running pip-compile requirements.in this will generate new requirements.txt file where all versions are pinned, it's clear, what is the origin (Optionally) you can run it with --generate-hashes option then you can (as with pipreqs) at any time create new environment based on this requirements.txt pip-tools offer you --upgrade option to upgrade the final reqs supports layered requirements (e.g. having dev and prod versions) there is integration with pre-commit offers pip-sync tool to update your environment based on requirements.txt There are few more stuff you can do with it and I really love the integration with pre-commit. This allows you to use the same requirements as before (just with .in suffix) and add pre-commit hook that automatically updates requirements.txt (so you will never experience having different local environment from the generated requirements.txt which might easily happen when you run something manually).


Instruction:
What is the difference between using loc and using just square brackets to filter for columns in Pandas/Python?

I've noticed three methods of selecting a column in a Pandas DataFrame: First method of selecting a column using loc: df_new = df.loc[:, 'col1'] Second method - seems simpler and faster: df_new = df['col1'] Third method - most convenient: df_new = df.col1 Is there a difference between these three methods? I don't think so, in which case I'd rather use the third method. I'm mostly curious as to why there appear to be three methods for doing the same thing.

Output: In the following situations, they behave the same: Selecting a single column (df['A'] is the same as df.loc[:, 'A'] -> selects column A) Selecting a list of columns (df[['A', 'B', 'C']] is the same as df.loc[:, ['A', 'B', 'C']] -> selects columns A, B and C) Slicing by rows (df[1:3] is the same as df.iloc[1:3] -> selects rows 1 and 2. Note, however, if you slice rows with loc, instead of iloc, you'll get rows 1, 2 and 3 assuming you have a RangeIndex. See details here.) However, [] does not work in the following situations: You can select a single row with df.loc[row_label] You can select a list of rows with df.loc[[row_label1, row_label2]] You can slice columns with df.loc[:, 'A':'C'] These three cannot be done with []. More importantly, if your selection involves both rows and columns, then assignment becomes problematic. df[1:3]['A'] = 5 This selects rows 1 and 2 then selects column 'A' of the returning object and assigns value 5 to it. The problem is, the returning object might be a copy so this may not change the actual DataFrame. This raises SettingWithCopyWarning. The correct way of making this assignment is: df.loc[1:3, 'A'] = 5 With .loc, you are guaranteed to modify the original DataFrame. It also allows you to slice columns (df.loc[:, 'C':'F']), select a single row (df.loc[5]), and select a list of rows (df.loc[[1, 2, 5]]). Also note that these two were not included in the API at the same time. .loc was added much later as a more powerful and explicit indexer. See unutbu's answer for more detail. Note: Getting columns with [] vs . is a completely different topic. . is only there for convenience. It only allows accessing columns whose names are valid Python identifiers (i.e. they cannot contain spaces, they cannot be composed of numbers...). It cannot be used when the names conflict with Series/DataFrame methods. It also cannot be used for non-existing columns (i.e. the assignment df.a = 1 won't work if there is no column a). Other than that, . and [] are the same.


Instruction:
Change a Django form field to a hidden field

I have a Django form with a RegexField, which is very similar to a normal text input field. In my view, under certain conditions I want to hide it from the user, and trying to keep the form as similar as possible. What's the best way to turn this field into a HiddenInput field? I know I can set attributes on the field with: form['fieldname'].field.widget.attr['readonly'] = 'readonly' And I can set the desired initial value with: form.initial['fieldname'] = 'mydesiredvalue' However, that won't change the form of the widget. What's the best / most "django-y" / least "hacky" way to make this field a <input type="hidden"> field?

Output: If you have a custom template and view you may exclude the field and use {{ modelform.instance.field }} to get the value. also you may prefer to use in the view: field = form.fields['field_name'] field.widget = field.hidden_widget() but I'm not sure it will protect save method on post. edit: field with multiple values don't supports HiddenInput as input type, so use default hidden input widget for this field instead.


Instruction:
cartesian product in pandas

I have two pandas dataframes: from pandas import DataFrame df1 = DataFrame({'col1':[1,2],'col2':[3,4]}) df2 = DataFrame({'col3':[5,6]}) What is the best practice to get their cartesian product (of course without writing it explicitly like me)? #df1, df2 cartesian product df_cartesian = DataFrame({'col1':[1,2,1,2],'col2':[3,4,3,4],'col3':[5,5,6,6]})

Output: In recent versions of Pandas (>= 1.2) this is built into merge so you can do: from pandas import DataFrame df1 = DataFrame({'col1':[1,2],'col2':[3,4]}) df2 = DataFrame({'col3':[5,6]}) df1.merge(df2, how='cross') This is equivalent to the previous pandas < 1.2 answer but is easier to read. For pandas < 1.2: If you have a key that is repeated for each row, then you can produce a cartesian product using merge (like you would in SQL). from pandas import DataFrame, merge df1 = DataFrame({'key':[1,1], 'col1':[1,2],'col2':[3,4]}) df2 = DataFrame({'key':[1,1], 'col3':[5,6]}) merge(df1, df2,on='key')[['col1', 'col2', 'col3']] Output: col1 col2 col3 0 1 3 5 1 1 3 6 2 2 4 5 3 2 4 6 See here for the documentation: http://pandas.pydata.org/pandas-docs/stable/merging.html


Instruction:
How to change values in a tuple?

I have a tuple called values which contains the following: ('275', '54000', '0.0', '5000.0', '0.0') I want to change the first value (i.e., 275) in this tuple but I understand that tuples are immutable so values[0] = 200 will not work. How can I achieve this?

Output: It's possible via: t = ('275', '54000', '0.0', '5000.0', '0.0') lst = list(t) lst[0] = '300' t = tuple(lst) But if you're going to need to change things, you probably are better off keeping it as a list


Instruction:
Renaming a virtualenv folder without breaking it

I've created folder and initialized a virtualenv instance in it. $ mkdir myproject $ cd myproject $ virtualenv env When I run (env)$ pip freeze, it shows the installed packages as it should. Now I want to rename myproject/ to project/. $ mv myproject/ project/ However, now when I run $ . env/bin/activate (env)$ pip freeze it says pip is not installed. How do I rename the project folder without breaking the environment?

Output: This answer uses an obsolete flag that is not available on virtualenv after v16.7.11 (2021-07-20) You need to adjust your install to use relative paths. virtualenv provides for this with the experimental --relocatable option. From the docs: Normally environments are tied to a specific path. That means that you cannot move an environment around or copy it to another computer. You can fix up an environment to make it relocatable with the command: $ virtualenv --relocatable ENV NOTE: ENV is the name of the virtual environment and you must run this from outside the ENV directory. This will make some of the files created by setuptools or distribute use relative paths, and will change all the scripts to use activate_this.py instead of using the location of the Python interpreter to select the environment. Note: you must run this after you've installed any packages into the environment. If you make an environment relocatable, then install a new package, you must run virtualenv --relocatable again.


Instruction:
Why are trailing commas allowed in a list?

I am curious why in Python a trailing comma in a list is valid syntax, and it seems that Python simply ignores it: >>> ['a','b',] ['a', 'b'] It makes sense when its a tuple since ('a') and ('a',) are two different things, but in lists?

Output: The main advantages are that it makes multi-line lists easier to edit and that it reduces clutter in diffs. Changing: s = ['manny', 'mo', 'jack', ] to: s = ['manny', 'mo', 'jack', 'roger', ] involves only a one-line change in the diff: s = ['manny', 'mo', 'jack', + 'roger', ] This beats the more confusing multi-line diff when the trailing comma was omitted: s = ['manny', 'mo', - 'jack' + 'jack', + 'roger' ] The latter diff makes it harder to see that only one line was added and that the other line didn't change content. It also reduces the risk of doing this: s = ['manny', 'mo', 'jack' 'roger' # Added this line, but forgot to add a comma on the previous line ] and triggering implicit string literal concatenation, producing s = ['manny', 'mo', 'jackroger'] instead of the intended result.


Instruction:
How do you do a simple "chmod +x" from within python?

I want to create a file from within a python script that is executable. import os import stat os.chmod('somefile', stat.S_IEXEC) it appears os.chmod doesn't 'add' permissions the way unix chmod does. With the last line commented out, the file has the filemode -rw-r--r--, with it not commented out, the file mode is ---x------. How can I just add the u+x flag while keeping the rest of the modes intact?

Output: Use os.stat() to get the current permissions, use | to OR the bits together, and use os.chmod() to set the updated permissions. Example: import os import stat st = os.stat('somefile') os.chmod('somefile', st.st_mode | stat.S_IEXEC)


Instruction:
How to get the PATH environment-variable separator in Python?

When multiple directories need to be concatenated, as in an executable search path, there is an os-dependent separator character. For Windows it's ';', for Linux it's ':'. Is there a way in Python to get which character to split on? In the discussions to this question How do I find out my python path using python? , it is suggested that os.sep will do it. That answer is wrong, since it is the separator for components of a directory or filename and equates to '\\' or '/'.

Output: os.pathsep The character conventionally used by the operating system to separate search path components (as in PATH), such as ':' for POSIX or ';' for Windows. Also available via os.path.


Instruction:
OpenAI API error 429: "You exceeded your current quota, please check your plan and billing details"

I'm making a Python script to use OpenAI via its API. However, I'm getting this error: openai.error.RateLimitError: You exceeded your current quota, please check your plan and billing details My script is the following: #!/usr/bin/env python3.8 # -*- coding: utf-8 -*- import openai openai.api_key = "<My PAI Key>" completion = openai.ChatCompletion.create( model="gpt-3.5-turbo", messages=[ {"role": "user", "content": "Tell the world about the ChatGPT API in the style of a pirate."} ] ) print(completion.choices[0].message.content) I'm declaring the shebang python3.8, because I'm using pyenv. I think it should work, since I did 0 API requests, so I'm assuming there's an error in my code.

Output: TL;DR: You need to upgrade to a paid plan. Set up a paid account, add a credit or debit card, and generate a new API key if your old one was generated before the upgrade. It might take 10 minutes or so after you upgrade to a paid plan before the paid account becomes active and the error disappears. Problem As stated in the official OpenAI documentation: TYPE OVERVIEW RateLimitError Cause: You have hit your assigned rate limit. Solution: Pace your requests. Read more in our rate limit guide. Also, read more about Error Code 429 - You exceeded your current quota, please check your plan and billing details: This (i.e., 429) error message indicates that you have hit your maximum monthly spend (hard limit) for the API. This means that you have consumed all the credits or units allocated to your plan and have reached the limit of your billing cycle. This could happen for several reasons, such as: You are using a high-volume or complex service that consumes a lot of credits or units per request. You are using a large or diverse data set that requires a lot of requests to process. Your limit is set too low for your organization’s usage. Did you sign up some time ago? You're getting error 429 because either you used all your free tokens or 3 months have passed since you signed up. As stated in the official OpenAI article: To explore and experiment with the API, all new users get $5 worth of free tokens. These tokens expire after 3 months. After the quota has passed you can choose to enter billing information to upgrade to a paid plan and continue your use of the API on pay-as-you-go basis. If no billing information is entered you will still have login access, but will be unable to make any further API requests. Please see the pricing page for the latest information on pay-as-you-go pricing. Note: If you signed up earlier (e.g., in December 2022), you got $18 worth of free tokens. Check your API usage in the usage dashboard. For example, my free trial expires tomorrow and this is what I see right now in the usage dashboard: This is how my dashboard looks after expiration: If I run a simple script after my free trial has expired, I get the following error: openai.error.RateLimitError: You exceeded your current quota, please check your plan and billing details. Did you create your second OpenAI account? You're getting error 429 because you created a second OpenAI account with the same phone number. It seems like free credit is given based on phone numbers. As explained on the official OpenAI forum by @SapphireFelineBytes: I created an Open AI account in November and my $18 credits expired on March 1st. So, like many of you here, I tried creating a new account with a different email address, but same number. They gave me $0 credits. I tried now with a different phone number and email. This time I got $5 credits. It's confirmed that free credit is given based on phone numbers, as explained on the official OpenAI forum by @logankilpatrick: Also note, you only get free credits for the first account associated with your phone number. Subsequent accounts are not granted free credits. Solution Try to do the following: Set up paid account. Add a credit or debit card. Generate a new API key if your old API key was generated before you upgraded to the paid plan. When you upgrade to a paid plan, don't expect the error to disappear immediately, as @dcferreira mentioned in the comment above. It might take a few minutes to more than an hour after the upgrade before the error disappears. In the comment below, @JoeMornin confirmed that it took 10 minutes for his paid account to become active. In the meantime, he was getting the following error: You've reached your usage limit. See your usage dashboard and billing settings for more details. If you have further questions, please contact us through our help center at help.openai.com.


Instruction:
AWS error from Python: No module named lambda_function

I am creating a AWS Lambda python deployment package. I am using one external dependency requests. I installed the external dependency using the AWS documentation. Below is my Python code. import requests print('Loading function') s3 = boto3.client('s3') def lambda_handler(event, context): #print("Received event: " + json.dumps(event, indent=2)) # Get the object from the event and show its content type bucket = event['Records'][0]['s3']['bucket']['name'] key = urllib.unquote_plus(event['Records'][0]['s3']['object']['key']).decode('utf8') try: response = s3.get_object(Bucket=bucket, Key=key) s3.download_file(bucket,key, '/tmp/data.txt') lines = [line.rstrip('\n') for line in open('/tmp/data.txt')] for line in lines: col=line.split(',') print(col[5],col[6]) print("CONTENT TYPE: " + response['ContentType']) return response['ContentType'] except Exception as e: print(e) print('Error getting object {} from bucket {}. Make sure they exist and your bucket is in the same region as this function.'.format(key, bucket)) raise e Created the Zip the content of the project-dir directory and uploaded to the lambda(Zip the directory content, not the directory). When I am execute the function I am getting the below mentioned error. START RequestId: 9e64e2c7-d0c3-11e5-b34e-75c7fb49d058 Version: $LATEST **Unable to import module 'lambda_function': No module named lambda_function** END RequestId: 9e64e2c7-d0c3-11e5-b34e-75c7fb49d058 REPORT RequestId: 9e64e2c7-d0c3-11e5-b34e-75c7fb49d058 Duration: 19.63 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 9 MB

Output: Error was due to file name of the lambda function. The console no longer asks you what handler name you want - it just sets it to "lambda_function.lambda_handler". So, you need to make sure your filename is "lambda_function.py" and the function name in there is "lambda_handler". The other alternative is explicitly setting the name of the handler after creating the function.


Instruction:
How to stop flask application without using ctrl-c

I want to implement a command which can stop flask application by using flask-script. I have searched the solution for a while. Because the framework doesn't provide app.stop() API, I am curious about how to code this. I am working on Ubuntu 12.10 and Python 2.7.3.

Output: If you are just running the server on your desktop, you can expose an endpoint to kill the server (read more at Shutdown The Simple Server): from flask import request def shutdown_server(): func = request.environ.get('werkzeug.server.shutdown') if func is None: raise RuntimeError('Not running with the Werkzeug Server') func() @app.get('/shutdown') def shutdown(): shutdown_server() return 'Server shutting down...' Here is another approach that is more contained: from multiprocessing import Process server = Process(target=app.run) server.start() # ... server.terminate() server.join() Let me know if this helps.


Instruction:
How to save traceback / sys.exc_info() values in a variable?

I want to save the name of the error and the traceback details into a variable. Here's is my attempt. import sys try: try: print x except Exception, ex: raise NameError except Exception, er: print "0", sys.exc_info()[0] print "1", sys.exc_info()[1] print "2", sys.exc_info()[2] Output: 0 <type 'exceptions.NameError'> 1 2 <traceback object at 0xbd5fc8> Desired Output: 0 NameError 1 2 Traceback (most recent call last): File "exception.py", line 6, in <module> raise NameError P.S. I know this can be done easily using the traceback module, but I want to know usage of sys.exc_info()[2] object here.

Output: This is how I do it: >>> import traceback >>> try: ... int('k') ... except: ... var = traceback.format_exc() ... >>> print(var) Traceback (most recent call last): File "<stdin>", line 2, in <module> ValueError: invalid literal for int() with base 10: 'k' You should however take a look at the traceback documentation, as you might find there more suitable methods, depending to how you want to process your variable afterwards...


Instruction:
What's the fastest way of checking if a point is inside a polygon in python

I found two main methods to look if a point belongs inside a polygon. One is using the ray tracing method used here, which is the most recommended answer, the other is using matplotlib path.contains_points (which seems a bit obscure to me). I will have to check lots of points continuously. Does anybody know if any of these two is more recommendable than the other or if there are even better third options? I checked the two methods and matplotlib looks much faster. from time import time import numpy as np import matplotlib.path as mpltPath # regular polygon for testing lenpoly = 100 polygon = [[np.sin(x)+0.5,np.cos(x)+0.5] for x in np.linspace(0,2*np.pi,lenpoly)[:-1]] # random points set of points to test N = 10000 points = np.random.rand(N,2) # Ray tracing def ray_tracing_method(x,y,poly): n = len(poly) inside = False p1x,p1y = poly[0] for i in range(n+1): p2x,p2y = poly[i % n] if y > min(p1y,p2y): if y <= max(p1y,p2y): if x <= max(p1x,p2x): if p1y != p2y: xints = (y-p1y)*(p2x-p1x)/(p2y-p1y)+p1x if p1x == p2x or x <= xints: inside = not inside p1x,p1y = p2x,p2y return inside start_time = time() inside1 = [ray_tracing_method(point[0], point[1], polygon) for point in points] print("Ray Tracing Elapsed time: " + str(time()-start_time)) # Matplotlib mplPath start_time = time() path = mpltPath.Path(polygon) inside2 = path.contains_points(points) print("Matplotlib contains_points Elapsed time: " + str(time()-start_time)) which gives, Ray Tracing Elapsed time: 0.441395998001 Matplotlib contains_points Elapsed time: 0.00994491577148 Same relative difference was obtained one using a triangle instead of the 100 sides polygon. I will also check shapely since it looks a package just devoted to these kind of problems.

Output: You can consider shapely: from shapely.geometry import Point from shapely.geometry.polygon import Polygon point = Point(0.5, 0.5) polygon = Polygon([(0, 0), (0, 1), (1, 1), (1, 0)]) print(polygon.contains(point)) From the methods you've mentioned I've only used the second, path.contains_points, and it works fine. In any case depending on the precision you need for your test I would suggest creating a numpy bool grid with all nodes inside the polygon to be True (False if not). If you are going to make a test for a lot of points this might be faster (although notice this relies you are making a test within a "pixel" tolerance): from matplotlib import path import matplotlib.pyplot as plt import numpy as np first = -3 size = (3-first)/100 xv,yv = np.meshgrid(np.linspace(-3,3,100),np.linspace(-3,3,100)) p = path.Path([(0,0), (0, 1), (1, 1), (1, 0)]) # square with legs length 1 and bottom left corner at the origin flags = p.contains_points(np.hstack((xv.flatten()[:,np.newaxis],yv.flatten()[:,np.newaxis]))) grid = np.zeros((101,101),dtype='bool') grid[((xv.flatten()-first)/size).astype('int'),((yv.flatten()-first)/size).astype('int')] = flags xi,yi = np.random.randint(-300,300,100)/100,np.random.randint(-300,300,100)/100 vflag = grid[((xi-first)/size).astype('int'),((yi-first)/size).astype('int')] plt.imshow(grid.T,origin='lower',interpolation='nearest',cmap='binary') plt.scatter(((xi-first)/size).astype('int'),((yi-first)/size).astype('int'),c=vflag,cmap='Greens',s=90) plt.show() The result is this:


Instruction:
Type hinting tuples in Python

When I want to typehint a tuple in Python like: def func(var: tuple[int, int]): # do something func((1, 2)) # would be fine func((1, 2, 3)) # would throw an error It is required to give the exact number of items in the tuple. That's different from list typehinting: def func(var: list[int]): # do something func([1]) # would be fine func([1, 2]) # would also be fine func([1, 2, 3]) # would also be fine That's consequentially, in a way, because of the type of tuples. Because they are designed not to be changed, you have to hardcode the amount of items in it. So my question is, is there a way to make the number of items in a tuple type hint flexible? I tried something like that but it didn't work: def func(var: tuple[*int]):

Output: Yes, you can make the number of items in a tuple type hint flexible. From Python 3.9 and on you can do: def func(var: tuple[int, ...]): pass Before Python 3.9 you would do: from typing import Tuple def func(var: Tuple[int, ...]): pass From the docs: https://docs.python.org/3/library/stdtypes.html#tuple To specify a variable-length tuple of homogeneous type, use literal ellipsis, e.g. tuple[int, ...]. A plain tuple is equivalent to tuple[Any, ...], and in turn to tuple.


Instruction:
How to create a "singleton" tuple with only one element

In the below example, I would expect all the elements to be tuples. Why is a tuple converted to a string when it only contains a single string? >>> a = [('a'), ('b'), ('c', 'd')] >>> a ['a', 'b', ('c', 'd')] >>> >>> for elem in a: ... print type(elem) ... <type 'str'> <type 'str'> <type 'tuple'>

Output: why is a tuple converted to a string when it only contains a single string? a = [('a'), ('b'), ('c', 'd')] Because those first two elements aren't tuples; they're just strings. The parenthesis don't automatically make them tuples. You have to add a comma after the string to indicate to python that it should be a tuple. >>> type( ('a') ) <type 'str'> >>> type( ('a',) ) <type 'tuple'> To fix your example code, add commas here: >>> a = [('a',), ('b',), ('c', 'd')] ^ ^ From the Python Docs: A special problem is the construction of tuples containing 0 or 1 items: the syntax has some extra quirks to accommodate these. Empty tuples are constructed by an empty pair of parentheses; a tuple with one item is constructed by following a value with a comma (it is not sufficient to enclose a single value in parentheses). Ugly, but effective. If you truly hate the trailing comma syntax, a workaround is to pass a list to the tuple() function: x = tuple(['a'])


Instruction:
How to set environment variables in PyCharm?

I have started to work on a Django project, and I would like to set some environment variables without setting them manually or having a bash file to source. I want to set the following variables: export DATABASE_URL=postgres://127.0.0.1:5432/my_db_name export DEBUG=1 # there are other variables, but they contain personal information I have read this, but that does not solve what I want. In addition, I have tried setting the environment variables in Preferences-> Build, Execution, Deployment->Console->Python Console/Django Console, but it sets the variables for the interpreter.

Output: You can set environmental variables in Pycharm's run configurations menu. Open the Run Configuration selector in the top-right and click Edit Configurations... Select the correct file from the menu, find Environmental variables and click ... Add or change variables, then click OK You can access your environmental variables with os.environ import os print(os.environ['SOME_VAR'])


Instruction:
ImportError: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with LibreSSL 2.8.3

After pip install openai, when I try to import openai, it shows this error: the 'ssl' module of urllib3 is compile with LibreSSL not OpenSSL I just followed a tutorial on a project about using API of OpenAI. But when I get to the first step which is the install and import OpenAI, I got stuck. And I tried to find the solution for this error but I found nothing. Here is the message after I try to import OpenAI: Python 3.9.6 (default, Mar 10 2023, 20:16:38) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import openai Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/yule/Library/Python/3.9/lib/python/site-packages/openai/__init__.py", line 19, in <module> from openai.api_resources import ( File "/Users/mic/Library/Python/3.9/lib/python/site-packages/openai/api_resources/__init__.py", line 1, in <module> from openai.api_resources.audio import Audio # noqa: F401 File "/Users/mic/Library/Python/3.9/lib/python/site-packages/openai/api_resources/audio.py", line 4, in <module> from openai import api_requestor, util File "/Users/mic/Library/Python/3.9/lib/python/site-packages/openai/api_requestor.py", line 22, in <module> import requests File "/Users/mic/Library/Python/3.9/lib/python/site-packages/requests/__init__.py", line 43, in <module> import urllib3 File "/Users/mic/Library/Python/3.9/lib/python/site-packages/urllib3/__init__.py", line 38, in <module> raise ImportError( ImportError: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with LibreSSL 2.8.3. See: https://github.com/urllib3/urllib3/issues/2168 I tried to --upgrade the urllib3, but it is still not working. The result is: pip3 install --upgrade urllib3 Defaulting to user installation because normal site-packages is not writeable Requirement already satisfied: urllib3 in ./Library/Python/3.9/lib/python/site-packages (2.0.2)

Output: The reason why the error message mentioned OpenSSL 1.1.1+ and LibreSSL 2.8.3 is that urllib3 v2.0 (the version you've installed) requires OpenSSL 1.1.1+ to work properly, as it relies on some new features of OpenSSL 1.1.1. The issue is that the version of the 'ssl' module that is currently installed in your environment is compiled with LibreSSL 2.8.3, which is not compatible with urllib3 v2.0. To use urllib3 v2.0, you need an 'ssl' module compiled with OpenSSL 1.1.1 or later, by trying: brew install openssl@1.1 Or you could use an older version of urllib3 that is compatible suc. For example urllib3 v1.26.6, which does not have a strict OpenSSL version requirement. You can force the version installing with this command: pip install urllib3==1.26.6


Instruction:
How can you set class attributes from variable arguments (kwargs) in python

Suppose I have a class with a constructor (or other function) that takes a variable number of arguments and then sets them as class attributes conditionally. I could set them manually, but it seems that variable parameters are common enough in python that there should be a common idiom for doing this. But I'm not sure how to do this dynamically. I have an example using eval, but that's hardly safe. I want to know the proper way to do this -- maybe with lambda? class Foo: def setAllManually(self, a=None, b=None, c=None): if a!=None: self.a = a if b!=None: self.b = b if c!=None: self.c = c def setAllWithEval(self, **kwargs): for key in **kwargs: if kwargs[param] != None eval("self." + key + "=" + kwargs[param])

Output: You could update the __dict__ attribute (which represents the instance attributes in the form of a dictionary) with the keyword arguments: class Bar(object): def __init__(self, **kwargs): self.__dict__.update(kwargs) then you can: >>> bar = Bar(a=1, b=2) >>> bar.a 1 and with something like: allowed_keys = {'a', 'b', 'c'} self.__dict__.update((k, v) for k, v in kwargs.items() if k in allowed_keys) you could filter the keys beforehand (use iteritems instead of items if you’re still using Python 2.x).


Instruction:
Python list in SQL query as parameter

I have a Python list, say l = [1,5,8] I want to write a SQL query to get the data for all the elements of the list, say select name from students where id = |IN THE LIST l| How do I accomplish this?

Output: Answers so far have been templating the values into a plain SQL string. That's absolutely fine for integers, but if we wanted to do it for strings we get the escaping issue. Here's a variant using a parameterised query that would work for both: placeholder= '?' # For SQLite. See DBAPI paramstyle. placeholders= ', '.join(placeholder for unused in l) query= 'SELECT name FROM students WHERE id IN (%s)' % placeholders cursor.execute(query, l)


Instruction:
Python group by

Assume that I have a set of data pair where index 0 is the value and index 1 is the type: input = [ ('11013331', 'KAT'), ('9085267', 'NOT'), ('5238761', 'ETH'), ('5349618', 'ETH'), ('11788544', 'NOT'), ('962142', 'ETH'), ('7795297', 'ETH'), ('7341464', 'ETH'), ('9843236', 'KAT'), ('5594916', 'ETH'), ('1550003', 'ETH') ] I want to group them by their type (by the 1st indexed string) as such: result = [ { 'type': 'KAT', 'items': ['11013331', '9843236'] }, { 'type': 'NOT', 'items': ['9085267', '11788544'] }, { 'type': 'ETH', 'items': ['5238761', '962142', '7795297', '7341464', '5594916', '1550003'] } ] How can I achieve this in an efficient way?

Output: Do it in 2 steps. First, create a dictionary. >>> input = [('11013331', 'KAT'), ('9085267', 'NOT'), ('5238761', 'ETH'), ('5349618', 'ETH'), ('11788544', 'NOT'), ('962142', 'ETH'), ('7795297', 'ETH'), ('7341464', 'ETH'), ('9843236', 'KAT'), ('5594916', 'ETH'), ('1550003', 'ETH')] >>> from collections import defaultdict >>> res = defaultdict(list) >>> for v, k in input: res[k].append(v) ... Then, convert that dictionary into the expected format. >>> [{'type':k, 'items':v} for k,v in res.items()] [{'items': ['9085267', '11788544'], 'type': 'NOT'}, {'items': ['5238761', '5349618', '962142', '7795297', '7341464', '5594916', '1550003'], 'type': 'ETH'}, {'items': ['11013331', '9843236'], 'type': 'KAT'}] It is also possible with itertools.groupby but it requires the input to be sorted first. >>> sorted_input = sorted(input, key=itemgetter(1)) >>> groups = groupby(sorted_input, key=itemgetter(1)) >>> [{'type':k, 'items':[x[0] for x in v]} for k, v in groups] [{'items': ['5238761', '5349618', '962142', '7795297', '7341464', '5594916', '1550003'], 'type': 'ETH'}, {'items': ['11013331', '9843236'], 'type': 'KAT'}, {'items': ['9085267', '11788544'], 'type': 'NOT'}] Note: before python 3.7, both of these do not respect the original order of the keys. You need an OrderedDict if you need to keep the order. >>> from collections import OrderedDict >>> res = OrderedDict() >>> for v, k in input: ... if k in res: res[k].append(v) ... else: res[k] = [v] ... >>> [{'type':k, 'items':v} for k,v in res.items()] [{'items': ['11013331', '9843236'], 'type': 'KAT'}, {'items': ['9085267', '11788544'], 'type': 'NOT'}, {'items': ['5238761', '5349618', '962142', '7795297', '7341464', '5594916', '1550003'], 'type': 'ETH'}] On or after python 3.7, a regular dict keeps insertion order.


Instruction:
Using Django time/date widgets in custom form

How can I use the nifty JavaScript date and time widgets that the default admin uses with my custom view? I have looked through the Django forms documentation, and it briefly mentions django.contrib.admin.widgets, but I don't know how to use it? Here is my template that I want it applied on. <form action="." method="POST">
 {% for f in form %}
  {{ f.name }}	 {{ f }}
 {% endfor %}
 <input type="submit" name="submit" value="Add Product"> </form> Also, I think it should be noted that I haven't really written a view up myself for this form, I am using a generic view. Here is the entry from the url.py: (r'^admin/products/add/$', create_object, {'model': Product, 'post_save_redirect': ''}), And I am relevantly new to the whole Django/MVC/MTV thing, so please go easy...

Output: The growing complexity of this answer over time, and the many hacks required, probably ought to caution you against doing this at all. It's relying on undocumented internal implementation details of the admin, is likely to break again in future versions of Django, and is no easier to implement than just finding another JS calendar widget and using that. That said, here's what you have to do if you're determined to make this work: Define your own ModelForm subclass for your model (best to put it in forms.py in your app), and tell it to use the AdminDateWidget / AdminTimeWidget / AdminSplitDateTime (replace 'mydate' etc with the proper field names from your model): from django import forms from my_app.models import Product from django.contrib.admin import widgets class ProductForm(forms.ModelForm): class Meta: model = Product def __init__(self, *args, **kwargs): super(ProductForm, self).__init__(*args, **kwargs) self.fields['mydate'].widget = widgets.AdminDateWidget() self.fields['mytime'].widget = widgets.AdminTimeWidget() self.fields['mydatetime'].widget = widgets.AdminSplitDateTime() Change your URLconf to pass 'form_class': ProductForm instead of 'model': Product to the generic create_object view (that'll mean from my_app.forms import ProductForm instead of from my_app.models import Product, of course). In the head of your template, include {{ form.media }} to output the links to the Javascript files. And the hacky part: the admin date/time widgets presume that the i18n JS stuff has been loaded, and also require core.js, but don't provide either one automatically. So in your template above {{ form.media }} you'll need: <script type="text/javascript" src="/my_admin/jsi18n/"></script> <script type="text/javascript" src="/media/admin/js/core.js"></script> You may also wish to use the following admin CSS (thanks Alex for mentioning this): <link rel="stylesheet" type="text/css" href="/media/admin/css/forms.css"/> <link rel="stylesheet" type="text/css" href="/media/admin/css/base.css"/> <link rel="stylesheet" type="text/css" href="/media/admin/css/global.css"/> <link rel="stylesheet" type="text/css" href="/media/admin/css/widgets.css"/> This implies that Django's admin media (ADMIN_MEDIA_PREFIX) is at /media/admin/ - you can change that for your setup. Ideally you'd use a context processor to pass this values to your template instead of hardcoding it, but that's beyond the scope of this question. This also requires that the URL /my_admin/jsi18n/ be manually wired up to the django.views.i18n.javascript_catalog view (or null_javascript_catalog if you aren't using I18N). You have to do this yourself instead of going through the admin application so it's accessible regardless of whether you're logged into the admin (thanks Jeremy for pointing this out). Sample code for your URLconf: (r'^my_admin/jsi18n', 'django.views.i18n.javascript_catalog'), Lastly, if you are using Django 1.2 or later, you need some additional code in your template to help the widgets find their media: {% load adminmedia %} /* At the top of the template. */ /* In the head section of the template. */ <script type="text/javascript"> window.__admin_media_prefix__ = "{% filter escapejs %}{% admin_media_prefix %}{% endfilter %}"; </script> Thanks lupefiasco for this addition.


Instruction:
ImportError: cannot import name '_unicodefun' from 'click'

When running our lint checks with the Python Black package, an error comes up: ImportError: cannot import name '_unicodefun' from 'click' (/Users/robot/.cache/pre-commit/repo3u71ccm2/py_env-python3.9/lib/python3.9/site-packages/click/init.py)` In researching this, I found the following related issues: ImportError: cannot import name '_unicodefun' from 'click' #2976 ImportError: cannot import name '_unicodefun' from 'click' #6013 How can I solve this problem? Is this a false positive from the linter? Do I need to modify my code?

Output: This has been fixed by Black 22.3.0. Versions before that won't work with click 8.1.0. Incompatible with click 8.1.0 (ImportError: cannot import name '_unicodefun' from 'click') #2964 E.g.: black.yml python-version: 3.8 - name: install black run: | - pip install black==20.8b1 + pip install black==22.3.0 - name: run black run: | black . --check --line-length 100 https://github.com/Clinical-Genomics/cgbeacon2/pull/221/files As a workaround, pin click to the last version via pip install --upgrade click==8.0.2.


Instruction:
How to convert a set to a list in python?

I am trying to convert a set to a list in Python 2.6. I'm using this syntax: first_list = [1,2,3,4] my_set=set(first_list) my_list = list(my_set) However, I get the following stack trace: Traceback (most recent call last): File "<console>", line 1, in <module> TypeError: 'set' object is not callable How can I fix this?

Output: It is already a list: >>> type(my_set) <class 'list'> Do you want something like: >>> my_set = set([1, 2, 3, 4]) >>> my_list = list(my_set) >>> print(my_list) [1, 2, 3, 4] EDIT: Output of your last comment: >>> my_list = [1,2,3,4] >>> my_set = set(my_list) >>> my_new_list = list(my_set) >>> print(my_new_list) [1, 2, 3, 4] I'm wondering if you did something like this: >>> set = set() >>> set([1, 2]) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: 'set' object is not callable


Instruction:
Good alternative to Pandas .append() method, now that it has been deprecated?

I use the following method a lot to append a single row to a dataframe. However, it has been deprecated. One thing I really like about it is that it allows you to append a simple dict object. For example: # Creating an empty dataframe df = pd.DataFrame(columns=['a', 'b']) # Appending a row df = df.append({ 'a': 1, 'b': 2 }, ignore_index=True) Again, what I like most about this is that the code is very clean and requires very few lines. Now I suppose the recommended alternative is: # Create the new row as its own dataframe df_new_row = pd.DataFrame({ 'a': [1], 'b': [2] }) df = pd.concat([df, df_new_row]) So what was one line of code before is now two lines with a throwaway variable and extra cruft where I create the new dataframe. :( Is there a good way to do this that just uses a dict like I have in the past (that is not deprecated)?

Output: Create a list with your dictionaries, if they are needed, and then create a new dataframe with df = pd.DataFrame.from_records(your_list). List's "append" method are very efficient and won't be ever deprecated. Dataframes on the other hand, frequently have to be recreated and all data copied over on appends, due to their design - that is why they deprecated the method