Instruction:
Removing duplicate characters from a string

How can I remove duplicate characters from a string using Python? For example, let's say I have a string: foo = 'mppmt' How can I make the string: foo = 'mpt' NOTE: Order is not important

Output: If order does not matter, you can use "".join(set(foo)) set() will create a set of unique letters in the string, and "".join() will join the letters back to a string in arbitrary order. If order does matter, you can use a dict instead of a set, which since Python 3.7 preserves the insertion order of the keys. (In the CPython implementation, this is already supported in Python 3.6 as an implementation detail.) foo = "mppmt" result = "".join(dict.fromkeys(foo)) resulting in the string "mpt". In earlier versions of Python, you can use collections.OrderedDict, which has been available starting from Python 2.7.


Instruction:
Python requests module sends JSON string instead of x-www-form-urlencoded param string

I was under the impression that POSTSs using x-www-form-urlencoded specifications should send a URL encoded param string in the body of the post. However, when I do this data = json.dumps({'param1': 'value1', 'param2': 'value2'}) Requests.post(url, data=data) The body of the request on the receiving end looks like this: {"param1": "value1", "param2": "value2"} But I was expecting to get this param1=value1&param2=value2 How I can get Requests to send the data in the second form?

Output: The reason you're getting JSON is because you're explicitly calling json.dumps to generate a JSON string. Just don't do that, and you won't get a JSON string. In other words, change your first line to this: data = {'param1': 'value1', 'param2': 'value2'} As the docs explain, if you pass a dict as the data value, it will be form-encoded, while if you pass a string, it will be sent as-is. For example, in one terminal window: $ nc -kl 8765 In another: $ python3 >>> import requests >>> d = {'spam': 20, 'eggs': 3} >>> requests.post("http://localhost:8765", data=d) ^C >>> import json >>> j = json.dumps(d) >>> requests.post("http://localhost:8765", data=j) ^C In the first terminal, you'll see that the first request body is this (and Content-Type application/x-www-form-urlencoded): spam=20&eggs=3 … while the second is this (and has no Content-Type): {"spam": 20, "eggs": 3}


Instruction:
GeoDjango GEOSException error

Trying to install a GeoDjango on my machine. I'm really new to Python and being brought into a project that has been a very tricky install for the other team members. I installed Python 2.7 and GEOS using brew, and running PSQL 9.2.4 but keep getting this error when I try to get the webserver running: __import__(name) File "/Users/armynante/Desktop/uclass-files/uclass-env/lib/python2.7/site packages/django/contrib/gis/geometry/backend/geos.py", line 1, in <module> from django.contrib.gis.geos import ( File "/Users/armynante/Desktop/uclass-files/uclass-env/lib/python2.7/site packages/django/contrib/gis/geos/__init__.py", line 6, in <module> from django.contrib.gis.geos.geometry import GEOSGeometry, wkt_regex, hex_regex File "/Users/armynante/Desktop/uclass-files/uclass-env/lib/python2.7/site packages/django/contrib/gis/geos/geometry.py", line 14, in <module> from django.contrib.gis.geos.coordseq import GEOSCoordSeq File "/Users/armynante/Desktop/uclass-files/uclass-env/lib/python2.7/site- packages/django/contrib/gis/geos/coordseq.py", line 9, in <module> from django.contrib.gis.geos.libgeos import CS_PTR File "/Users/armynante/Desktop/uclass-files/uclass-env/lib/python2.7/site- packages/django/contrib/gis/geos/libgeos.py", line 119, in <module> _verinfo = geos_version_info() File "/Users/armynante/Desktop/uclass-files/uclass-env/lib/python2.7/site packages/django/contrib/gis/geos/libgeos.py", line 115, in geos_version_info if not m: raise GEOSException('Could not parse version info string "%s"' % ver) django.contrib.gis.geos.error.GEOSException: Could not parse version info string "3.4.2-CAPI-1.8.2 r3921" Cant seem to find anything relevant to this trace on SO or the web. I think it might be a regex failure? I'm currently trying to reinstall PSQL and GEOS to see if I can get it running. Here is my requirements file: django==1.4 psycopg2==2.4.4 py-bcrypt==0.4 python-memcached==1.48 south==0.7.3 # Debug Tools sqlparse==0.1.3 django-debug-toolbar==0.9.1 django-devserver==0.3.1 # Deployment fabric==1.4 # AWS # boto==2.1.1 django-storages==1.1.4 django-ses==0.4.1 # ECL http://packages.elmcitylabs.com/ecl_django-0.5.3.tar.gz#ecl_django http://packages.elmcitylabs.com/ecl_google-0.2.14.tar.gz#ecl_google # https://packages.elmcitylabs.com/ecl_tools-0.3.7.tar.gz#ecl_tools # https://packages.elmcitylabs.com/chargemaster-0.2.19.tar.gz # https://packages.elmcitylabs.com/ecl_facebook-0.3.12.tar.gz#ecl_facebook # https://packages.elmcitylabs.com/ecl_twitter-0.3.3.tar.gz#ecl_twitter # Search #https://github.com/elmcitylabs/django-haystack/tarball/issue-522#django-haystack -e git+https://github.com/toastdriven/django-haystack.git#egg=django-haystack pysolr==2.1.0-beta # whoosh==2.3.2 # Misc # PIL # django-shorturls==1.0.1 # suds==0.4 django-mptt sorl-thumbnail stripe pytz==2013b

Output: This is my solution (obviously it is ugly, like my English, but works). The problem is that the versions string has an white space unwanted in the RegEx. The error says: GEOSException: Could not parse version info string "3.4.2-CAPI-1.8.2 r3921" And the geos_version_info warns: Regular expression should be able to parse version strings such as '3.0.0rc4-CAPI-1.3.3', '3.0.0-CAPI-1.4.1' or '3.4.0dev-CAPI-1.8.0' Edit this file: site-packages/django/contrib/gis/geos/libgeos.py Look for the function: geos_version_info And change this line: ver = geos_version().decode() With this line: ver = geos_version().decode().split(' ')[0] There is also another problem, where there is a whitespace at the end but no more information is provided. Such version also doesn't match version regular expression, so strip()-ping the version may be expected behaviour as a quick fix. In my example it was: '3.8.0-CAPI-1.13.1 '


Instruction:
Accessing an attribute using a variable in Python

How do I reference this_prize.left or this_prize.right using a variable? from collections import namedtuple import random Prize = namedtuple("Prize", ["left", "right"]) this_prize = Prize("FirstPrize", "SecondPrize") if random.random() > .5: choice = "left" else: choice = "right" # retrieve the value of "left" or "right" depending on the choice print("You won", this_prize.choice) AttributeError: 'Prize' object has no attribute 'choice'

Output: The expression this_prize.choice is telling the interpreter that you want to access an attribute of this_prize with the name "choice". But this attribute does not exist in this_prize. What you actually want is to return the attribute of this_prize identified by the value of choice. So you just need to change your last line using the getattr() method... from collections import namedtuple import random Prize = namedtuple("Prize", ["left", "right" ]) this_prize = Prize("FirstPrize", "SecondPrize") if random.random() > .5: choice = "left" else: choice = "right" # retrieve the value of "left" or "right" depending on the choice print "You won", getattr(this_prize, choice)


Instruction:
Plotting CDF of a pandas series in python

Is there a way to do this? I cannot seem an easy way to interface pandas series with plotting a CDF (cumulative distribution function).

Output: In case you are also interested in the values, not just the plot. import pandas as pd # If you are in jupyter %matplotlib inline This will always work (discrete and continuous distributions) # Define your series s = pd.Series([9, 5, 3, 5, 5, 4, 6, 5, 5, 8, 7], name = 'value') df = pd.DataFrame(s) # Get the frequency, PDF and CDF for each value in the series # Frequency stats_df = df \ .groupby('value') \ ['value'] \ .agg('count') \ .pipe(pd.DataFrame) \ .rename(columns = {'value': 'frequency'}) # PDF stats_df['pdf'] = stats_df['frequency'] / sum(stats_df['frequency']) # CDF stats_df['cdf'] = stats_df['pdf'].cumsum() stats_df = stats_df.reset_index() stats_df # Plot the discrete Probability Mass Function and CDF. # Technically, the 'pdf label in the legend and the table the should be 'pmf' # (Probability Mass Function) since the distribution is discrete. # If you don't have too many values / usually discrete case stats_df.plot.bar(x = 'value', y = ['pdf', 'cdf'], grid = True) Alternative example with a sample drawn from a continuous distribution or you have a lot of individual values: # Define your series s = pd.Series(np.random.normal(loc = 10, scale = 0.1, size = 1000), name = 'value') # ... all the same calculation stuff to get the frequency, PDF, CDF # Plot stats_df.plot(x = 'value', y = ['pdf', 'cdf'], grid = True) For continuous distributions only Please note if it is very reasonable to make the assumption that there is only one occurence of each value in the sample (typically encountered in the case of continuous distributions) then the groupby() + agg('count') is not necessary (since the count is always 1). In this case, a percent rank can be used to get to the cdf directly. Use your best judgment when taking this kind of shortcut! :) # Define your series s = pd.Series(np.random.normal(loc = 10, scale = 0.1, size = 1000), name = 'value') df = pd.DataFrame(s) # Get to the CDF directly df['cdf'] = df.rank(method = 'average', pct = True) # Sort and plot df.sort_values('value').plot(x = 'value', y = 'cdf', grid = True)


Instruction:
How to disable TOKENIZERS_PARALLELISM=(true | false) warning?

I use pytorch to train huggingface-transformers model, but every epoch, always output the warning: The current process just got forked. Disabling parallelism to avoid deadlocks... To disable this warning, please explicitly set TOKENIZERS_PARALLELISM=(true | false) How to disable this warning?

Output: Set the environment variable to the string "false" either by TOKENIZERS_PARALLELISM=false in your shell or by: import os os.environ["TOKENIZERS_PARALLELISM"] = "false" in the Python script


Instruction:
PySpark create new column with mapping from a dict

Using Spark 1.6, I have a Spark DataFrame column (named let's say col1) with values A, B, C, DS, DNS, E, F, G and H. I want to create a new column (say col2) with the values from the dict here below. How do I map this? (e,g. 'A' needs to be mapped to 'S' etc.) dict = {'A': 'S', 'B': 'S', 'C': 'S', 'DS': 'S', 'DNS': 'S', 'E': 'NS', 'F': 'NS', 'G': 'NS', 'H': 'NS'}

Output: Inefficient solution with UDF (version independent): from pyspark.sql.types import StringType from pyspark.sql.functions import udf def translate(mapping): def translate_(col): return mapping.get(col) return udf(translate_, StringType()) df = sc.parallelize([('DS', ), ('G', ), ('INVALID', )]).toDF(['key']) mapping = { 'A': 'S', 'B': 'S', 'C': 'S', 'DS': 'S', 'DNS': 'S', 'E': 'NS', 'F': 'NS', 'G': 'NS', 'H': 'NS'} df.withColumn("value", translate(mapping)("key")) with the result: +-------+-----+ | key|value| +-------+-----+ | DS| S| | G| NS| |INVALID| null| +-------+-----+ Much more efficient (Spark >= 2.0, Spark < 3.0) is to create a MapType literal: from pyspark.sql.functions import col, create_map, lit from itertools import chain mapping_expr = create_map([lit(x) for x in chain(*mapping.items())]) df.withColumn("value", mapping_expr.getItem(col("key"))) with the same result: +-------+-----+ | key|value| +-------+-----+ | DS| S| | G| NS| |INVALID| null| +-------+-----+ but more efficient execution plan: == Physical Plan == *Project [key#15, keys: [B,DNS,DS,F,E,H,C,G,A], values: [S,S,S,NS,NS,NS,S,NS,S][key#15] AS value#53] +- Scan ExistingRDD[key#15] compared to UDF version: == Physical Plan == *Project [key#15, pythonUDF0#61 AS value#57] +- BatchEvalPython [translate_(key#15)], [key#15, pythonUDF0#61] +- Scan ExistingRDD[key#15] In Spark >= 3.0 getItem should be replaced with __getitem__ ([]), i.e: from pyspark.sql.functions import col, create_map, lit from itertools import chain mapping_expr = create_map([lit(x) for x in chain(*mapping.items())]) df.withColumn("value", mapping_expr[col("key")])


Instruction:
Python: best practice and securest way to connect to MySQL and execute queries

What is the safest way to run queries on MySQL? I am aware of the dangers involved with MySQL and SQL injection. However, I do not know how I should run my queries to prevent injection on the variables to which other users (webclients) can manipulate. I used to write my own escape function, but apparently this is "not-done". What should I use and how should I use it to query and do inserts safely on a MySQL database through python without risking MySQL injection?

Output: To avoid injections, use execute with %s in place of each variable, then pass the value via a list or tuple as the second parameter of execute. Here is an example from the documentation: c=db.cursor() max_price=5 c.execute("""SELECT spam, eggs, sausage FROM breakfast WHERE price < %s""", (max_price,)) Note that this is using a comma, not % (which would be a direct string substitution, not escaped). Don't do this: c.execute("""SELECT spam, eggs, sausage FROM breakfast WHERE price < %s""" % (max_price,)) In addition, you must not use single quotes around the position holder ('%s') if the parameter is a string as the driver provides these.


Instruction:
ImageMagick not authorized to convert PDF to an image

I have a program, in which I need to convert a PDF to an image using Image Magick. I do that using the subprocess package: cmd = 'magick convert -density 300 '+pdfFile+'['+str(rangeTuple[0])+'-'+str(rangeTuple[1])+'] -depth 8 '+'temp.tiff' #WINDOWS if(os.path.isfile('temp.tiff')): os.remove('temp.tiff') subprocess.call(cmd,shell=True) im = Image.open('temp.tiff') The error I got is: convert-im6.q16: not authorized `temp2.pdf' @ error/constitute.c/ReadImage/412. convert-im6.q16: no images defined `temp.tiff' @ error/convert.c/ConvertImageCommand/3258. Traceback (most recent call last): File "UKExtraction2.py", line 855, in <module> doItAllUpper("A0","UK5.csv","temp",59,70,"box",2,1000,firstPageCoordsUK,boxCoordUK,voterBoxCoordUK,internalBoxNumberCoordUK,externalBoxNumberCoordUK,addListInfoUK) File "UKExtraction2.py", line 776, in doItAllUpper doItAll(tempPDFName,outputCSV,2,pdfs,formatType,n_blocks,writeBlockSize,firstPageCoords,boxCoord,voterBoxCoord,internalBoxNumberCoord,externalBoxNumberCoord,addListInfo,pdfName) File "UKExtraction2.py", line 617, in doItAll mainProcess(pdfName,(0,noOfPages-1),formatType,n_blocks,outputCSV,writeBlockSize,firstPageCoords,boxCoord,voterBoxCoord,internalBoxNumberCoord,externalBoxNumberCoord,addListInfo,bigPDFName,basePages) File "UKExtraction2.py", line 542, in mainProcess im = Image.open('temp.tiff') File "/home/rohit/.local/lib/python3.6/site-packages/PIL/Image.py", line 2609, in open fp = builtins.open(filename, "rb") FileNotFoundError: [Errno 2] No such file or directory: 'temp.tiff' The most important of which is: convert-im6.q16: not authorized `temp2.pdf' @ error/constitute.c/ReadImage/412. I think this is because ImageMagick isn't authorized to access the PDF. What should be done now? I'm on a Linux server. Any help is appreciated.

Output: emcconville is correct. More specifically edit the Imagemagick policy.xml file to uncomment this line: <!-- <policy domain="module" rights="none" pattern="{PS,PDF,XPS}" /> --> And change it from rights="none" to rights="read|write" <policy domain="module" rights="read|write" pattern="{PS,PDF,XPS}" /> This was a recent addition to the policy.xml file, I believe, due to a security flaw found in the Ghostscript delegate. I think that flaw has now been fixed in the current version of Ghostscript, which is 9.25. NOTE: On some systems the policy line will have domain="coder" rather than domain="module"


Instruction:
Print raw HTTP request in Flask or WSGI

I am debugging a microcontroller I've built which is writing raw HTTP requests line by line. I am using Flask for my backend and I would like to see the entire request as it appears in this format: GET / HTTP/1.1 Content-length: 123 User-agent: blah ... I know Flask is based on WSGI. Is there anyway to get this to work with Flask?

Output: With flask you have access to the request object which contains all the HTTP details: from flask import request @app.route('/') def index(): print(request.headers)


Instruction:
How to monkeypatch python's datetime.datetime.now with py.test?

I need to test functions which uses datetime.datetime.now(). What is the easiest way to do this?

Output: You need to monkeypatch datetime.now function. In example below, I'm creating fixture which I can re-use later in other tests: import datetime import pytest FAKE_TIME = datetime.datetime(2020, 12, 25, 17, 5, 55) @pytest.fixture def patch_datetime_now(monkeypatch): class mydatetime(datetime.datetime): @classmethod def now(cls): return FAKE_TIME monkeypatch.setattr(datetime, 'datetime', mydatetime) def test_patch_datetime(patch_datetime_now): assert datetime.datetime.now() == FAKE_TIME


Instruction:
Python-redis keys() returns list of bytes objects instead of strings

I'm using the regular redis package in order to connect my Python code to my Redis server. As part of my code I check if a string object is existed in my Redis server keys. string = 'abcde' if string in redis.keys(): do something.. For some reasons, redis.keys() returns a list with bytes objects, such as [b'abcde'], while my string is, of course, a str object. I already tried to set charset, encoding and decode_responses in my redis generator, but it did not help. My goal is to insert the data as string ahead, and not iterate over the keys list and change each element to str() while checking it. Thanks ahead

Output: You can configure the Redis client to automatically convert responses from bytes to strings using the decode_responses argument to the StrictRedis constructor: r = redis.StrictRedis('localhost', 6379, charset="utf-8", decode_responses=True) Make sure you are consistent with the charset option between clients. Note You would be better off using the EXISTS command and restructuring your code like: string = 'abcde' if redis.exists(string): do something.. The KEYS operation returns every key in your Redis database and will cause serious performance degradation in production. As a side effect you avoid having to deal with the binary to string conversion.


Instruction:
Iterating over Numpy matrix rows to apply a function each?

I want to be able to iterate over the matrix to apply a function to each row. How can I do it for a Numpy matrix ?

Output: You can use numpy.apply_along_axis(). Assuming that your array is 2D, you can use it like: import numpy as np myarray = np.array([[11, 12, 13], [21, 22, 23], [31, 32, 33]]) def myfunction(x): return x[0] + x[1]**2 + x[2]**3 print(np.apply_along_axis(myfunction, axis=1, arr=myarray)) #[ 2352 12672 36992]


Instruction:
"synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'." problem in TensorFlow

I installed TensorFlow 1.10.1 but when I tried to import TensorFlow it said that I need TensorFlow version 1.10.0. Thus, I installed it and now I get the following warnings: >>> import tensorflow C:\Users\PC\Anaconda3\envs\tut\lib\site-packages\tensorflow\python\framework\dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint8 = np.dtype([("qint8", np.int8, 1)]) C:\Users\PC\Anaconda3\envs\tut\lib\site-packages\tensorflow\python\framework\dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint8 = np.dtype([("quint8", np.uint8, 1)]) C:\Users\PC\Anaconda3\envs\tut\lib\site-packages\tensorflow\python\framework\dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint16 = np.dtype([("qint16", np.int16, 1)]) C:\Users\PC\Anaconda3\envs\tut\lib\site-packages\tensorflow\python\framework\dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint16 = np.dtype([("quint16", np.uint16, 1)]) C:\Users\PC\Anaconda3\envs\tut\lib\site-packages\tensorflow\python\framework\dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint32 = np.dtype([("qint32", np.int32, 1)]) C:\Users\PC\Anaconda3\envs\tut\lib\site-packages\tensorflow\python\framework\dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. np_resource = np.dtype([("resource", np.ubyte, 1)]) C:\Users\PC\Anaconda3\envs\tut\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint8 = np.dtype([("qint8", np.int8, 1)]) C:\Users\PC\Anaconda3\envs\tut\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint8 = np.dtype([("quint8", np.uint8, 1)]) C:\Users\PC\Anaconda3\envs\tut\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint16 = np.dtype([("qint16", np.int16, 1)]) C:\Users\PC\Anaconda3\envs\tut\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint16 = np.dtype([("quint16", np.uint16, 1)]) C:\Users\PC\Anaconda3\envs\tut\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint32 = np.dtype([("qint32", np.int32, 1)]) C:\Users\PC\Anaconda3\envs\tut\lib\site-packages\tensorboard\compat\tensorflow_stub\dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. np_resource = np.dtype([("resource", np.ubyte, 1)])

Output: It just a warning, not an error. It occurring because your current numpy libray version is not compatible with tensorflow version. You need to downgrade numpy version. tensorflow 1.10.0 has requirement numpy<=1.14.5,>=1.13.3, but you must have some higher version installed(this warning message occurs with newest numpy version 1.17.0).


Instruction:
Overriding properties in python

So, I'm trying to figure out the best (most elegant with the least amount of code) way to allow overriding specific functions of a property (e.g., just the getter, just the setter, etc.) in python. I'm a fan of the following way of doing properties, due to the fact that all of their methods are encapsulated in the same indented block of code (it's easier to see where the functions dealing with one property stop and the functions dealing with the next begin): @apply def foo(): """A foobar""" def fget(self): return self._foo def fset(self, val): self._foo = val return property(**locals()) However, if I want to inherit from a class that defines properties in this manner, and then, say, override the foo setter function, it seems tricky. I've done some searching and most of the answers I've found have been to define separate functions in the base class (e.g. getFoo and setFoo), explicitly create a property definition from them (e.g. foo = property(lambda x: x.getFoo(), lambda x, y: x.setFoo(y), lambda x: x.delFoo())), and then override getFoo, setFoo, and delFoo as needed. I dislike this solution because it means I have to define lambas for every single property, and then write out each function call (when before I could have just done property(**locals())). I also don't get the encapsulation that I had originally. Ideally, what I would like to be able to do would be something like this: class A(object): def __init__(self): self.foo = 8 @apply def foo(): """A foobar""" def fget(self): return self._foo def fset(self, val): self._foo = val return property(**locals()) class ATimesTwo(A): @some_decorator def foo(): def fset(self, val): self._foo = val * 2 return something And then the output would look something like: >>> a = A() >>> a.foo 8 >>> b = ATimesTwo() >>> b.foo 16 Basically, ATimesTwo inherits the getter function from A but overrides the setter function. Does anybody know of a way to do this (in a manner that looks similar to the example above)? What function would the some_decorator look like, and what should the foo function return?

Output: I'm sure you've heard this before, but apply has been deprecated for eight years, since Python 2.3. Don't use it. Your use of locals() is also contrary to the Zen of Python -- explicit is better than implicit. If you really like the increased indentation, there is no need to create a throwaway object, just do if True: @property def foo(self): return self._foo @foo.setter def foo(self, val): self._foo = val Which doesn't abuse locals, use apply, require creation of an extra object, or need a line afterwards with foo = foo() making it harder to see the end of the block. It works just as well for your old-fashioned way of using property -- just do foo = property(fget, fset) as normal. If you want to override a property in an arbitrary subclass, you can use a recipe like this. If the subclass knows where the property was defined, just do: class ATimesTwo(A): @A.foo.setter def foo(self, val): self._foo = val * 2


Instruction:
How to change filehandle with Python logging on the fly with different classes and imports

I cannot perform an on-the-fly logging fileHandle change. For example, I have 3 classes one.py import logging class One(): def __init__(self,txt="?"): logging.debug("Hey, I'm the class One and I say: %s" % txt) two.py import logging class Two(): def __init__(self,txt="?"): logging.debug("Hey, I'm the class Two and I say: %s" % txt) config.py import logging class Config(): def __init__(self,logfile=None): logging.debug("Reading config") self.logfile(logfile) myapp from one import One from two import Two from config import Config import logging #Set default logging logging.basicConfig( level=logging.getLevelName(DEBUG), format='%(asctime)s - %(name)s - %(levelname)s - %(message)s', filename=None ) logging.info("Starting with stdout") o=One(txt="STDOUT") c=Config(logfile="/tmp/logfile") # Here must be the code that change the logging configuration and set the filehandler t=One(txt="This must be on the file, not STDOUT") If I try loggin.basicConfig() again, it doesn't work.

Output: Indeed, logging.basicConfig does nothing if a handler has been set up already: This function does nothing if the root logger already has handlers configured, unless the keyword argument force is set to True. You'll need to either add force=True (requires Python 3.8 or newer), or, alternatively, replace the current handler on the root logger: import logging fileh = logging.FileHandler('/tmp/logfile', 'a') formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s') fileh.setFormatter(formatter) log = logging.getLogger() # root logger for hdlr in log.handlers[:]: # remove all old handlers log.removeHandler(hdlr) log.addHandler(fileh) # set the new handler See the Configuring Logging chapter in the Python Logging HOWTO.


Instruction:
df.unique() on whole DataFrame based on a column

I have a DataFrame df filled with rows and columns where there are duplicate Id's: Index Id Type 0 a1 A 1 a2 A 2 b1 B 3 b3 B 4 a1 A ... When I use: uniqueId = df["Id"].unique() I get a list of unique IDs. How can I apply this filtering on the whole DataFrame such that it keeps the structure but that the duplicates (based on "Id") are removed?

Output: It seems you need DataFrame.drop_duplicates with parameter subset which specify where are test duplicates: #keep first duplicate value df = df.drop_duplicates(subset=['Id']) print (df) Id Type Index 0 a1 A 1 a2 A 2 b1 B 3 b3 B #keep last duplicate value df = df.drop_duplicates(subset=['Id'], keep='last') print (df) Id Type Index 1 a2 A 2 b1 B 3 b3 B 4 a1 A #remove all duplicate values df = df.drop_duplicates(subset=['Id'], keep=False) print (df) Id Type Index 1 a2 A 2 b1 B 3 b3 B


Instruction:
string representation of a numpy array with commas separating its elements

I have a numpy array, for example: points = np.array([[-468.927, -11.299, 76.271, -536.723], [-429.379, -694.915, -214.689, 745.763], [ 0., 0., 0., 0. ]]) if I print it or turn it into a string with str() I get: print w_points [[-468.927 -11.299 76.271 -536.723] [-429.379 -694.915 -214.689 745.763] [ 0. 0. 0. 0. ]] I need to turn it into a string that prints with separating commas while keeping the 2D array structure, that is: [[-468.927, -11.299, 76.271, -536.723], [-429.379, -694.915, -214.689, 745.763], [ 0., 0., 0., 0. ]] Does anybody know an easy way of turning a numpy array to that form of string? I know that .tolist() adds the commas but the result loses the 2D structure.

Output: Try using repr >>> import numpy as np >>> points = np.array([[-468.927, -11.299, 76.271, -536.723], ... [-429.379, -694.915, -214.689, 745.763], ... [ 0., 0., 0., 0. ]]) >>> print(repr(points)) array([[-468.927, -11.299, 76.271, -536.723], [-429.379, -694.915, -214.689, 745.763], [ 0. , 0. , 0. , 0. ]]) If you plan on using large numpy arrays, set np.set_printoptions(threshold=np.nan) first. Without it, the array representation will be truncated after about 1000 entries (by default). >>> arr = np.arange(1001) >>> print(repr(arr)) array([ 0, 1, 2, ..., 998, 999, 1000]) Of course, if you have arrays that large, this starts to become less useful and you should probably analyze the data some way other than just looking at it and there are better ways of persisting a numpy array than saving it's repr to a file...


Instruction:
How to properly use python's isinstance() to check if a variable is a number?

I found some old Python code that was doing something like: if type(var) is type(1): ... As expected, pep8 complains about this recommending usage of isinstance(). Now, the problem is that the numbers module was added in Python 2.6 and I need to write code that works with Python 2.5+ So if isinstance(var, Numbers.number) is not a solution. Which would be the proper solution in this case?

Output: In Python 2, you can use the types module: >>> import types >>> var = 1 >>> NumberTypes = (types.IntType, types.LongType, types.FloatType, types.ComplexType) >>> isinstance(var, NumberTypes) True Note the use of a tuple to test against multiple types. Under the hood, IntType is just an alias for int, etc.: >>> isinstance(var, (int, long, float, complex)) True The complex type requires that your python was compiled with support for complex numbers; if you want to guard for this use a try/except block: >>> try: ... NumberTypes = (types.IntType, types.LongType, types.FloatType, types.ComplexType) ... except AttributeError: ... # No support for complex numbers compiled ... NumberTypes = (types.IntType, types.LongType, types.FloatType) ... or if you just use the types directly: >>> try: ... NumberTypes = (int, long, float, complex) ... except NameError: ... # No support for complex numbers compiled ... NumberTypes = (int, long, float) ... In Python 3 types no longer has any standard type aliases, complex is always enabled and there is no longer a long vs int difference, so in Python 3 always use: NumberTypes = (int, float, complex) Last but not least, you can use the numbers.Numbers abstract base type (new in Python 2.6) to also support custom numeric types that don't derive directly from the above types: >>> import numbers >>> isinstance(var, numbers.Number) True This check also returns True for decimal.Decimal() and fractions.Fraction() objects. This module does make the assumption that the complex type is enabled; you'll get an import error if it is not.


Instruction:
Unexpected Exception: name 'basestring' is not defined when invoking ansible2

I'm trying to execute ansible2 commands... When I do: ansible-playbook -vvv -i my/inventory my/playbook.yml I get: Unexpected Exception: name 'basestring' is not defined the full traceback was: Traceback (most recent call last): File "/usr/local/bin/ansible-playbook", line 85, in <module> sys.exit(cli.run()) File "/usr/local/lib/python3.4/site-packages/ansible/cli/playbook.py", line 150, in run results = pbex.run() File "/usr/local/lib/python3.4/site-packages/ansible/executor/playbook_executor.py", line 87, in run self._tqm.load_callbacks() File "/usr/local/lib/python3.4/site-packages/ansible/executor/task_queue_manager.py", line 149, in load_callbacks elif isinstance(self._stdout_callback, basestring): NameError: name 'basestring' is not defined Here is ansible --version: ansible 2.0.0.2 config file = /etc/ansible/ansible.cfg configured module search path = Default w/o overrides And here is python --version Python 3.4.3

Output: Ansible below version 2.5 requires Python 2.6 or 2.7 on the control host: Control Node Requirements basestring is no longer available in Python 3. From What’s New In Python 3.0: The builtin basestring abstract type was removed. Use str instead. The str and bytes types don’t have functionality enough in common to warrant a shared base class. The 2to3 tool (see below) replaces every occurrence of basestring with str. So the solution is to either upgrade Ansible or downgrade Python.


Instruction:
Python BeautifulSoup give multiple tags to findAll

I'm looking for a way to use findAll to get two tags, in the order they appear on the page. Currently I have: import requests import BeautifulSoup def get_soup(url): request = requests.get(url) page = request.text soup = BeautifulSoup(page) get_tags = soup.findAll('hr' and 'strong') for each in get_tags: print each If I use that on a page with only 'em' or 'strong' in it then it will get me all of those tags, if I use on one with both it will get 'strong' tags. Is there a way to do this? My main concern is preserving the order in which the tags are found.

Output: You could pass a list, to find any of the given tags: tags = soup.find_all(['hr', 'strong'])


Instruction:
Automatically play sound in IPython notebook

I often run long-running cells in my IPython notebook. I'd like the notebook to automatically beep or play a sound when the cell is finished executing. Is there some way to do this in iPython notebook, or maybe some command I can put at the end of a cell that will automatically play a sound? I'm using Chrome if that makes any difference.

Output: TL;DR At the top of your notebook from IPython.display import Audio sound_file = './sound/beep.wav' sound_file should point to a file on your computer, or accessible from the internet. Then later, at the end of the long-running cell <code that takes a long time> Audio(sound_file, autoplay=True) This method uses the Audio tag built into Newer versions of iPython/Jupyter. Note For Older Versions Older versions without the Audio tag can use the following method. Put this in a cell and run it before you want to play your sound: from IPython.display import HTML from base64 import b64encode path_to_audio = "/path/to/snd/my-sound.mp3" audio_type = "mp3" sound = open(path_to_audio, "rb").read() sound_encoded = b64encode(sound) sound_tag = """ <audio id="beep" controls src="data:audio/{1};base64,{0}"> </audio>""".format(sound_encoded, audio_type) play_beep = """ <script type="text/javascript"> var audio = document.getElementById("beep"); audio.play(); </script> """ HTML(sound_tag) At the end of the cell you want to make a noise on completion put this: HTML(play_beep) How it works: It reads a file from the filesystem using iPython's built in open and read methods. Then it encodes this into base64. It then creates an audio tag with the ID beep and injects the base64 data into it. The final piece of setup creates a small script tag that plays the sound. This method should work in any browser that supports the HTML5 audio tag. Note: if you'd rather not display the audio controls in your notebook, just remove the controls attribute from the variable named sound_tag


Instruction:
Replace all occurrences of a string in a pandas dataframe (Python)

I have a pandas dataframe with about 20 columns. It is possible to replace all occurrences of a string (here a newline) by manually writing all column names: df['columnname1'] = df['columnname1'].str.replace("\n","
") df['columnname2'] = df['columnname2'].str.replace("\n","
") df['columnname3'] = df['columnname3'].str.replace("\n","
") ... df['columnname20'] = df['columnname20'].str.replace("\n","
") This unfortunately does not work: df = df.replace("\n","
") Is there any other, more elegant solution?

Output:
You can use replace and pass the strings to find/replace as dictionary keys/items: df.replace({'\n': '
'}, regex=True) For example: >>> df = pd.DataFrame({'a': ['1\n', '2\n', '3'], 'b': ['4\n', '5', '6\n']}) >>> df a b 0 1\n 4\n 1 2\n 5 2 3 6\n >>> df.replace({'\n': '
'}, regex=True) a b 0 1
 4
 1 2
 5 2 3 6
 Note that this method returns a new DataFrame instance by default (it does not modify the original), so you'll need to either reassign the output: df = df.replace({'\n': '
'}, regex=True) or specify inplace=True: df.replace({'\n': '
'}, regex=True, inplace=True)


Instruction:
sqlalchemy existing database query

I am using SQLAlchemy as ORM for a python project. I have created few models/schema and it is working fine. Now I need to query a existing MySQL database, no insert/update just the select statement. How can I create a wrapper around the tables of this existing database? I have briefly gone through the sqlalchemy docs and SO but couldn't find anything relevant. All suggest execute method, where I need to write the raw sql queries, while I want to use the SQLAlchemy query method in same way as I am using with the SA models. For example if the existing db has table name User then I want to query it using the dbsession ( only the select operation, probably with join)

Output: You seem to have an impression that SQLAlchemy can only work with a database structure created by SQLAlchemy (probably using MetaData.create_all()) - this is not correct. SQLAlchemy can work perfectly with a pre-existing database, you just need to define your models to match database tables. One way to do that is to use reflection, as Ilja Everilä suggests: from sqlalchemy import Table from sqlalchemy.orm import DeclarativeBase class Base(DeclarativeBase): pass class MyClass(Base): __table__ = Table('mytable', Base.metadata, autoload=True, autoload_with=some_engine) (which, in my opinion, would be totally fine for one-off scripts but may lead to incredibly frustrating bugs in a "real" application if there's a potential that the database structure may change over time) Another way is to simply define your models as usual taking care to define your models to match the database tables, which is not that difficult. The benefit of this approach is that you can map only a subset of database tables to you models and even only a subset of table columns to your model's fields. Suppose you have 10 tables in the database but only interested in users table from where you only need id, name and email fields: import sqlalchemy as sa from sqlalchemy.orm import DeclarativeBase class Base(DeclarativeBase): pass class User(Base): id = sa.Column(sa.Integer, primary_key=True) name = sa.Column(sa.String) email = sa.Column(sa.String) (note how we didn't need to define some details which are only needed to emit correct DDL, such as the length of the String fields or the fact that the email field has an index) SQLAlchemy will not emit INSERT/UPDATE queries unless you create or modify models in your code. If you want to ensure that your queries are read-only you may create a special user in the database and grant that user SELECT privileges only. Alternatively/in addition, you may also experiment with rolling back the transaction in your application code.


Instruction:
convert a grayscale image to a 3-channel image

I want to convert a gray-scale image with shape (height,width) to a 3 channels image with shape (height,width,nchannels). The work is done with a for-loop, but there must be a neat way. Here is a piece code in program, can someone give a hint. please advice. 30 if img.shape == (height,width): # if img is grayscale, expand 31 print "convert 1-channel image to ", nchannels, " image." 32 new_img = np.zeros((height,width,nchannels)) 33 for ch in range(nchannels): 34 for xx in range(height): 35 for yy in range(width): 36 new_img[xx,yy,ch] = img[xx,yy] 37 img = new_img

Output: You can use np.stack to accomplish this much more concisely: img = np.array([[1, 2], [3, 4]]) stacked_img = np.stack((img,)*3, axis=-1) print(stacked_img) # array([[[1, 1, 1], # [2, 2, 2]], # [[3, 3, 3], # [4, 4, 4]]])


Instruction:
Pandas : Reading first n rows from parquet file?

I have a parquet file and I want to read first n rows from the file into a pandas data frame. What I tried: df = pd.read_parquet(path= 'filepath', nrows = 10) It did not work and gave me error: TypeError: read_table() got an unexpected keyword argument 'nrows' I did try the skiprows argument as well but that also gave me same error. Alternatively, I can read the complete parquet file and filter the first n rows, but that will require more computations which I want to avoid. Is there any way to achieve it?

Output: The accepted answer is out of date. It is now possible to read only the first few lines of a parquet file into pandas, though it is a bit messy and backend dependent. To read using PyArrow as the backend, follow below: from pyarrow.parquet import ParquetFile import pyarrow as pa pf = ParquetFile('file_name.pq') first_ten_rows = next(pf.iter_batches(batch_size = 10)) df = pa.Table.from_batches([first_ten_rows]).to_pandas() Change the line batch_size = 10 to match however many rows you want to read in.


Instruction:
Is there a quiet version of subprocess.call?

Is there a variant of subprocess.call that can run the command without printing to standard out, or a way to block out it's standard out messages?

Output: Yes. Redirect its stdout to /dev/null. process = subprocess.call(["my", "command"], stdout=open(os.devnull, 'wb'))


Instruction:
How to access the last element in a Pandas series

Let us consider the following data frame: import pandas as pd d = {'col1': [1, 2, 3], 'col2': [3, 4, 5]} df=pd.DataFrame(data=d) If I want to access the first element in pandas series df['col1'], I can simply go df['col1'][0]. But how can I access the last element in this series? I have tried df['col1'][-1] which returns the following error: KeyError: -1L I know that I could go for something like df['col1'][len(df)-1] but why is reverse indexing impossible here?

Output: For select last value need Series.iloc or Series.iat, because df['col1'] return Series: print (df['col1'].iloc[-1]) 3 print (df['col1'].iat[-1]) 3 Or convert Series to numpy array and select last: print (df['col1'].values[-1]) 3 Or use DataFrame.iloc or DataFrame.iat - but is necessary position of column by Index.get_loc: print (df.iloc[-1, df.columns.get_loc('col1')]) 3 print (df.iat[-1, df.columns.get_loc('col1')]) 3 Or is possible use last value of index (necessary not duplicated) and select by DataFrame.loc: print (df.loc[df.index[-1], 'col1']) 3


Instruction:
Pandas GroupBy and select rows with the minimum value in a specific column

I have a DataFrame with columns A, B, and C. For each value of A, I would like to select the row with the minimum value in column B. That is, from this: df = pd.DataFrame({'A': [1, 1, 1, 2, 2, 2], 'B': [4, 5, 2, 7, 4, 6], 'C': [3, 4, 10, 2, 4, 6]}) A B C 0 1 4 3 1 1 5 4 2 1 2 10 3 2 7 2 4 2 4 4 5 2 6 6 I would like to get: A B C 0 1 2 10 1 2 4 4 For the moment I am grouping by column A, then creating a value that indicates to me the rows I will keep: a = data.groupby('A').min() a['A'] = a.index to_keep = [str(x[0]) + str(x[1]) for x in a[['A', 'B']].values] data['id'] = data['A'].astype(str) + data['B'].astype('str') data[data['id'].isin(to_keep)] I am sure that there is a much more straightforward way to do this. I have seen many answers here that use MultiIndex, which I would prefer to avoid. Thank you for your help.

Output: I feel like you're overthinking this. Just use groupby and idxmin: df.loc[df.groupby('A').B.idxmin()] A B C 2 1 2 10 4 2 4 4 df.loc[df.groupby('A').B.idxmin()].reset_index(drop=True) A B C 0 1 2 10 1 2 4 4


Instruction:
Django: dependencies reference nonexistent parent node

When I run the following command python manage.py migrate I receive this error from django so can't step forward in my practice: Traceback (most recent call last): File "manage.py", line 10, in <module> execute_from_command_line(sys.argv) File "/home/nikhil/testWeb-devEnv/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 385, in execute_from_command_line utility.execute() File "/home/nikhil/testWeb-devEnv/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 377, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/home/nikhil/testWeb-devEnv/local/lib/python2.7/site-packages/django/core/management/base.py", line 288, in run_from_argv self.execute(*args, **options.__dict__) File "/home/nikhil/testWeb-devEnv/local/lib/python2.7/site-packages/django/core/management/base.py", line 338, in execute output = self.handle(*args, **options) File "/home/nikhil/testWeb-devEnv/local/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 63, in handle executor = MigrationExecutor(connection, self.migration_progress_callback) File "/home/nikhil/testWeb-devEnv/local/lib/python2.7/site-packages/django/db/migrations/executor.py", line 17, in __init__ self.loader = MigrationLoader(self.connection) File "/home/nikhil/testWeb-devEnv/local/lib/python2.7/site-packages/django/db/migrations/loader.py", line 48, in __init__ self.build_graph() File "/home/nikhil/testWeb-devEnv/local/lib/python2.7/site-packages/django/db/migrations/loader.py", line 241, in build_graph self.graph.add_dependency(migration, key, parent) File "/home/nikhil/testWeb-devEnv/local/lib/python2.7/site-packages/django/db/migrations/graph.py", line 42, in add_dependency raise KeyError("Migration %s dependencies reference nonexistent parent node %r" % (migration, parent)) KeyError: u"Migration testBolt.0001_initial dependencies reference nonexistent parent node (u'delivery_boy', u'0004_auto_20150221_2011')" How do I solve this problem?

Output: Solution - 1 Remove pyc files from your migrations folder. Solution - 2 Need to remove that reference from testBolt.0001_initial by editing migration file. Solution - 3 Remove the new changes from the models and run python manage.py migrate --fake Now again modify your models with new changes Run python manage.py makemigrations And then again run python manage.py migrate


Instruction:
How to create a tuple of an empty tuple in Python?

How can I create a tuple consisting of just an empty tuple, i.e. (())? I have tried tuple(tuple()), tuple(tuple(tuple())), tuple([]) and tuple(tuple([])) which all gave me (). The reason that I use such a thing is as follows: Assume you have n bags with m items. To represent a list of items in a bag, I use a tuple of length n where each element of that tuple is a representative for a bag. A bag might be empty, which is labeled by (). Now, at some initial point, I have just one bag with empty items!

Output: The empty tuple is () (or the more-verbose and slower tuple()), and a tuple with just one item (such as the integer 1), called a singleton (see here and here) is (1,). Therefore, the tuple containing only the empty tuple is ((),) Here are some results showing that works: >>> a=((),) >>> type(a) <type 'tuple'> >>> len(a) 1 >>> a[0] () >>> type(a[0]) <type 'tuple'> >>> len(a[0]) 0


Instruction:
What's the correct way to clean up after an interrupted event loop?

I have an event loop that runs some co-routines as part of a command line tool. The user may interrupt the tool with the usual Ctrl + C, at which point I want to clean up properly after the interrupted event loop. Here's what I tried. import asyncio @asyncio.coroutine def shleepy_time(seconds): print("Shleeping for {s} seconds...".format(s=seconds)) yield from asyncio.sleep(seconds) if __name__ == '__main__': loop = asyncio.get_event_loop() # Side note: Apparently, async() will be deprecated in 3.4.4. # See: https://docs.python.org/3.4/library/asyncio-task.html#asyncio.async tasks = [ asyncio.async(shleepy_time(seconds=5)), asyncio.async(shleepy_time(seconds=10)) ] try: loop.run_until_complete(asyncio.gather(*tasks)) except KeyboardInterrupt as e: print("Caught keyboard interrupt. Canceling tasks...") # This doesn't seem to be the correct solution. for t in tasks: t.cancel() finally: loop.close() Running this and hitting Ctrl + C yields: $ python3 asyncio-keyboardinterrupt-example.py Shleeping for 5 seconds... Shleeping for 10 seconds... ^CCaught keyboard interrupt. Canceling tasks... Task was destroyed but it is pending! task: <Task pending coro=<shleepy_time() running at asyncio-keyboardinterrupt-example.py:7> wait_for=<Future cancelled> cb=[gather.<locals>._done_callback(1)() at /usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/asyncio/tasks.py:587]> Task was destroyed but it is pending! task: <Task pending coro=<shleepy_time() running at asyncio-keyboardinterrupt-example.py:7> wait_for=<Future cancelled> cb=[gather.<locals>._done_callback(0)() at /usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/asyncio/tasks.py:587]> Clearly, I didn't clean up correctly. I thought perhaps calling cancel() on the tasks would be the way to do it. What's the correct way to clean up after an interrupted event loop?

Output: If using Python 3.7+, using asyncio.run to execute your main function will take care of this, as pointed out in this answer. See below for an explanation of why just calling t.cancel() with the code in the question does not work, and a way to handle it that was relevant on older versions of Python, prior to the introduction of asyncio.run(). When you CTRL+C, the event loop gets stopped, so your calls to t.cancel() don't actually take effect. For the tasks to be cancelled, you need to start the loop back up again. Here's how you can handle it: import asyncio @asyncio.coroutine def shleepy_time(seconds): print("Shleeping for {s} seconds...".format(s=seconds)) yield from asyncio.sleep(seconds) if __name__ == '__main__': loop = asyncio.get_event_loop() # Side note: Apparently, async() will be deprecated in 3.4.4. # See: https://docs.python.org/3.4/library/asyncio-task.html#asyncio.async tasks = asyncio.gather( asyncio.async(shleepy_time(seconds=5)), asyncio.async(shleepy_time(seconds=10)) ) try: loop.run_until_complete(tasks) except KeyboardInterrupt as e: print("Caught keyboard interrupt. Canceling tasks...") tasks.cancel() loop.run_forever() tasks.exception() finally: loop.close() Once we catch KeyboardInterrupt, we call tasks.cancel() and then start the loop up again. run_forever will actually exit as soon as tasks gets cancelled (note that cancelling the Future returned by asyncio.gather also cancels all the Futures inside of it), because the interrupted loop.run_until_complete call added a done_callback to tasks that stops the loop. So, when we cancel tasks, that callback fires, and the loop stops. At that point we call tasks.exception, just to avoid getting a warning about not fetching the exception from the _GatheringFuture.


Instruction:
How do I get the name of the rows from the index of a data frame?

Consider a data frame with row names that aren't a column of their own per se, such as the following: X Y Row 1 0 5 Row 2 8 1 Row 3 3 0 How would I extract the name of these rows as a list, if I have their index? For example, it would look something like: function_name(dataframe[indices]) > ['Row 1', 'Row 2']

Output: df.index outputs the row names as pandas Index object. list(df.index) casts to a list. df.index['Row 2':'Row 5'] supports label slicing similar to columns.


Instruction:
Unexpected '{' in field name when doing string formatting

I'm trying to write a small script that will automate some PHP boilerplate that I need to write. It should write a copy of the string code to the output file with the various replacement fields filled in for each dict in the fields list. However, I'm getting the error: Traceback (most recent call last): File "writefields.py", line 43, in <module> formatted = code.format(**field) ValueError: unexpected '{' in field name As far as I can tell, there are no extra braces in either the replacement fields or the dicts that should be causing issues, so any help would be appreciated. code = ''' // {label} add_filter( 'submit_job_form_fields', 'frontend_add_{fieldname}_field' ); function frontend_add_{fieldname}_field($fields) { $fields['job']['job_{fieldname}'] = array( 'label' => __('{label}', 'job_manager'), 'type' => 'text', 'required' => {required}, 'priority' => 7, 'placeholder' => '{placeholder}' ); return $fields; } add_filter( 'job_manager_job_listing_data_fields', 'admin_add_{fieldname}_field' ); function admin_add_{fieldname}_field( $fields ) { $fields['_job_{fieldname}'] = array( 'label' => __( '{label}', 'job_manager' ), 'type' => 'text', 'placeholder' => '{placeholder}', 'description' => '' ); return $fields; } ''' fields = [ { 'fieldname': 'salary', 'label': 'Salary ($)', 'required': 'true', 'placeholder': 'e.g. 20000', }, { 'fieldname': 'test', 'label': 'Test Field', 'required': 'true', 'placeholder': '', } ] with open('field-out.txt', 'w') as f: for field in fields: formatted = code.format(**field) f.write(formatted) f.write('\n')

Output: You need to double any { or } that are not part of a formatting placeholder. For example, you have: function admin_add_{fieldname}_field( $fields ) { [....] } in the string. The { at the end of the first line and } on the last are not part of a placeholder. Replace them with {{ and }} respectively: function admin_add_{fieldname}_field( $fields ) {{ [....] }} Doubling up those curly braces escapes them; the final output will contain single { and } characters again. In the full string you used, that'd be: code = ''' // {label} add_filter( 'submit_job_form_fields', 'frontend_add_{fieldname}_field' ); function frontend_add_{fieldname}_field($fields) {{ $fields['job']['job_{fieldname}'] = array( 'label' => __('{label}', 'job_manager'), 'type' => 'text', 'required' => {required}, 'priority' => 7, 'placeholder' => '{placeholder}' ); return $fields; }} add_filter( 'job_manager_job_listing_data_fields', 'admin_add_{fieldname}_field' ); function admin_add_{fieldname}_field( $fields ) {{ $fields['_job_{fieldname}'] = array( 'label' => __( '{label}', 'job_manager' ), 'type' => 'text', 'placeholder' => '{placeholder}', 'description' => '' ); return $fields; }} '''


Instruction:
How to parse json file with c-style comments?

I have a json file, such as the following: { "author":"John", "desc": "If it is important to decode all valid JSON correctly \ and speed isn't as important, you can use the built-in json module, \ orsimplejson. They are basically the same but sometimes simplej \ further along than the version of it that is included with \ distribution." //"birthday": "nothing" //I comment this line } This file is auto created by another program. How do I parse it with Python?

Output: I can not imagine a json file "auto created by other program" would contain comments inside. Because json spec defines no comment at all, and that is by design, so no json library would output a json file with comment. Those comments are usually added later, by a human. No exception in this case. The OP mentioned that in his post: //"birthday": "nothing" //I comment this line. So the real question should be, how do I properly comment some content in a json file, yet maintaining its compliance with spec and hence its compatibility with other json libraries? And the answer is, rename your field to another name. Example: { "foo": "content for foo", "bar": "content for bar" } can be changed into: { "foo": "content for foo", "this_is_bar_but_been_commented_out": "content for bar" } This will work just fine most of the time because the consumer will very likely ignore unexpected fields (but not always, it depends on your json file consumer's implementation. So YMMV.) UPDATE: Apparently some reader was unhappy because this answer does not give the "solution" they expect. Well, in fact, I did give a working solution, by implicitly linking to the JSON designer's quote: Douglas Crockford Public Apr 30, 2012 Comments in JSON I removed comments from JSON because I saw people were using them to hold parsing directives, a practice which would have destroyed interoperability. I know that the lack of comments makes some people sad, but it shouldn't. Suppose you are using JSON to keep configuration files, which you would like to annotate. Go ahead and insert all the comments you like. Then pipe it through JSMin before handing it to your JSON parser. So, yeah, go ahead to use JSMin. Just keep in mind that when you are heading towards "using comments in JSON", that is a conceptually uncharted territory. There is no guarantee that whatever tools you choose would handle: inline [1,2,3,/* a comment */ 10], Python style [1, 2, 3] # a comment (which is a comment in Python but not in Javascript), INI style [1, 2, 3] ; a comment, ..., you get the idea. I would still suggest to NOT adding noncompliant comments in JSON in the first place.


Instruction:
Getting affected row count from psycopg2 connection.commit()

Currently, I have the following method to execute INSERT/UPDATE/DELETE statements using psycopg2 in Python: def exec_statement(_cxn, _stmt): try: db_crsr = _cxn.cursor() db_crsr.execute(_stmt) _cxn.commit() db_crsr.close() return True except: return False But what I would really like it to do, instead of bool, is return the row count affected by the transaction or -1 if the operation fails. Is there a way to get a number of rows affected by _cxn.commit()? E.g. for a single INSERT it would be always 1, for a DELETE or UPDATE, the number of rows affected by the statement etc.?

Output: commit() can't be used to get the row count, but you can use the cursor to get that information after each execute call. You can use its rowcount attribute to get the number of rows affected for SELECT, INSERT, UPDATE and DELETE. i.e. db_crsr = _cxn.cursor() db_crsr.execute(_stmt) rowcount = db_crsr.rowcount _cxn.commit() db_crsr.close() return rowcount If you want to return the number of affected rows, I would recommend not catching any exceptions, since if the operation truly failed (say the query was malformed, or there was a FK constraint violation, etc.), an exception should be raised, and in that case the caller could catch that and behave as desired. (Or, if you want to centralize the exception handling, perhaps raise a custom MyPostgresException, or similar.) -1 can be returned in a non-failure case in certain situations (https://www.psycopg.org/docs/cursor.html#cursor.rowcount), so I would recommend against using that value as the failure indicator. If you really want to return a numerical value in the case of failure, perhaps returning a number like -10 would work (in the except block), since rowcount shouldn't ever return that.


Instruction:
Why does mypy think library imports are missing?

When I run mypy it complains that modules cannot be found: sal@ahfang:~/workspace/ecs/cx-project-skeleton-repo/src/cx-example-function$ pipenv run python -m mypy . example_lambda.py:3: error: Cannot find module named 'aws_xray_sdk.core' But when trying to import that exact same module with the exact same Python interpreter, it seems that the module does exist and is importable. python Python 3.7.3 (default, Apr 3 2019, 05:39:12) [GCC 8.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import aws_xray_sdk.core >>> Other than to force ignore the imports in the mypy.ini file, is there anything I should be doing to help mypy see importable modules that definitely do exist?

Output: So, here's the crux of the issue: mypy does not try type-checking every single module you've imported. Instead, it only attempts to type-check modules that have explicitly opted-in to the typing ecosystem. Modules can opt-in to the typing ecosystem via two key mechanisms: Add type hints or stubs to their code, and include a file named py.typed within the package they distribute to PyPi (or any other package repository). The presence of this marker makes the package PEP-561-aware. The mypy docs also have more info about PEP-561-aware packages. Alternatively, add stubs to typeshed, the repository of type hints for the standard library and select 3rd party libraries. The aws_xray_sdk package has done neither of these things, so will be ignored by mypy. This is a bit unfortunate, so what can you do? The Missing imports section of the mypy docs has some detailed recommendations on what to do, but to summarize, you basically have three options which I'll list in order from least to most effort: Just silence the import by manually add # type: ignore comments to each import. You can also add the following section to your mypy config file to have this happen automatically: [mypy-aws_xray_sdk] ignore_missing_imports = True Now, anything you import from this module will be treated as being of type Any. Search around and see if anybody has created a third party stubs package for your library: basically, an unofficial (or sometimes semi-official) PEP-561-aware package that only contains type hints. For example, for django, there's django-stubs, for SqlAlchemy, there's sqlalchemy-stubs. Create your own stubs for this library and point to them via the mypy_path option in your mypy config file: mypy_path = my_stubs/aws_xray_sdk, my_stubs/some_other_library These stubs don't have to be complete, necessarily: you can get away with just adding annotations for the few things you're using. (And if they do end up becoming relatively complete, you perhaps look into contributing them back to the open-source community.) Now finally, you may be wondering why mypy behaves this way? Part of this is because it's not safe in the general case for mypy to just try finding and analyzing the module. Just blindly importing and using packages that are not type-hinting ready can sometimes result in odd type errors, or worse, can result in code incorrectly being marked as type-safe. That is, if you care about type-safety, it's better to be immediately notified that some package you're using has no type hints instead of mypy blindly inferring and smearing Anys all over your code. Mypy could give a better error message here though, at least most of the time. IMO the fact that it doesn't is largely an oversight. There's some discussion about this in https://github.com/python/mypy/issues/4542.


Instruction:
in ipython how do I accept and use an autocomplete suggestion?

I'm using Python 3.8.9 with IPython 8.0.1 on macOS. When I type anything whatsoever, it displays a predicted suggestion based on past commands. Cool. However, how do I actually accept that suggestion? I tried the obvious: tab, which does not accept the suggestion, but rather opens up a menu with different suggestions, while the original suggestion is still there (see screenshot). I also tried space, and return, but both of those act as if the suggestion was never made. How the heck do I actually use the ipython autosuggestion? Or is tab supposed to work and something is wrong with my ipython build or something?

Output: CTRL-E, CTRL-F, or Right Arrow Key https://ipython.readthedocs.io/en/8.13.2/config/shortcuts/index.html Alternatively, the End key, as suggested by Richard Berg (below).


Instruction:
What's the exact usage of __reduce__ in Pickler

I know that in order to be picklable, a class has to overwrite __reduce__ method, and it has to return string or tuple. How does this function work? What the exact usage of __reduce__? When will it been used?

Output: When you try to pickle an object, there might be some properties that don't serialize well. One example of this is an open file handle. Pickle won't know how to handle the object and will throw an error. You can tell the pickle module how to handle these types of objects natively within a class directly. Lets see an example of an object which has a single property; an open file handle: import pickle class Test(object): def __init__(self, file_path="test1234567890.txt"): # An open file in write mode self.some_file_i_have_opened = open(file_path, 'wb') my_test = Test() # Now, watch what happens when we try to pickle this object: pickle.dumps(my_test) It should fail and give a traceback: Traceback (most recent call last): File "<stdin>", line 1, in <module> --- snip snip a lot of lines --- File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy_reg.py", line 70, in _reduce_ex raise TypeError, "can't pickle %s objects" % base.__name__ TypeError: can't pickle file objects However, had we defined a __reduce__ method in our Test class, pickle would have known how to serialize this object: import pickle class Test(object): def __init__(self, file_path="test1234567890.txt"): # Used later in __reduce__ self._file_name_we_opened = file_path # An open file in write mode self.some_file_i_have_opened = open(self._file_name_we_opened, 'wb') def __reduce__(self): # we return a tuple of class_name to call, # and optional parameters to pass when re-creating return (self.__class__, (self._file_name_we_opened, )) my_test = Test() saved_object = pickle.dumps(my_test) # Just print the representation of the string of the object, # because it contains newlines. print(repr(saved_object)) This should give you something like: "c__main__\nTest\np0\n(S'test1234567890.txt'\np1\ntp2\nRp3\n.", which can be used to recreate the object with open file handles: print(vars(pickle.loads(saved_object))) In general, the __reduce__ method needs to return a tuple with at least two elements: A blank object class to call. In this case, self.__class__ A tuple of arguments to pass to the class constructor. In the example it's a single string, which is the path to the file to open. Consult the docs for a detailed explanation of what else the __reduce__ method can return.


Instruction:
Apply function on each row (row-wise) of a NumPy array

So, I have the function - def function(x): x , y = vector return exp(((-x**2/200))-0.5*(y+0.05*(x**2) - 100*0.05)**2) and let's say that I would like to evaluate it at the following points (first column are the x-values and second column are the y-values) - array([[-1.56113514, 4.51759732], [-2.80261623, 5.068371 ], [ 0.7792729 , 6.0169462 ], [-1.35672858, 3.52517478], [-1.92074891, 5.79966161], [-2.79340321, 4.73430001], [-2.79655868, 5.05361163], [-2.13637747, 5.39255837], [ 0.17341809, 3.60918261], [-1.22712921, 4.95327158]]) i.e. I would like to pass the function the first row of values and evaluate, then the second row and evaluate etc. and then the final result would be an array of the values evaluated at these points (so, an array consisting of 10 values). So, for example, if the function was, say, a bivariate normal distribution - def function2(x): function2 = (mvnorm.pdf(x,[0,0],[[1,0],[0,1]])) return function2 and I passed the above values into this function, I would get - array([ 1.17738907e-05, 1.08383957e-04, 1.69855078e-04, 5.64757613e-06, 1.37432346e-05, 1.44032800e-04, 1.33426313e-05, 1.97822328e-06, 6.56121709e-08, 4.67076770e-05]) So basically, I am looking for a way to rewrite the function so that it can do this. Moreover, I would like to keep the function as a function of one variable only (i.e. only a function of x). Thank you for your help!

Output: You can use np.apply_along_axis: np.apply_along_axis(function, 1, array) The first argument is the function, the second argument is the axis along which the function is to be applied. In your case, it is the first axis. The last argument is the array, of course. You should be warned, however, that apply_along_axis is only a convenience function, not a magic bullet. It has a severe speed limitation, since it just hides a loop. You should always try to vectorize your computation, where possible. Here's how I'd do this: v = array[:, 0] ** 2 # computing just once return np.exp((-v / 200) - 0.5 * (array[:, 1] + 0.05 * v - 5) ** 2)


Instruction:
How to apply default value to Python dataclass field when None was passed?

I need a class that will accept a number of parameters, I know that all parameters will be provided but some maybe passed as None in which case my class will have to provide default values. I want to setup a simple dataclass with a some default values like so: @dataclass class Specs1: a: str b: str = 'Bravo' c: str = 'Charlie' I would like to be able to get the default value for the second field but still set a value for the third one. I cannot do this with None because it is happily accepted as a value for my string: r1 = Specs1('Apple', None, 'Cherry') # Specs1(a='Apple', b=None, c='Cherry') I have come up with the following solution: @dataclass class Specs2: def_b: ClassVar = 'Bravo' def_c: ClassVar = 'Charlie' a: str b: str = def_b c: str = def_c def __post_init__(self): self.b = self.def_b if self.b is None else self.b self.c = self.def_c if self.c is None else self.c Which seems to behave as intended: r2 = Specs2('Apple', None, 'Cherry') # Specs2(a='Apple', b='Bravo', c='Cherry') However, I feel it is quite ugly and that I am maybe missing something here. My actual class will have more fields so it will only get uglier. The parameters passed to the class contain None and I do not have control over this aspect.

Output: I know this is a little late, but inspired by MikeSchneeberger's answer I made a small adaptation to the __post_init__ function that allows you to keep the defaults in the standard format: from dataclasses import dataclass, fields def __post_init__(self): # Loop through the fields for field in fields(self): # If there is a default and the value of the field is none we can assign a value if not isinstance(field.default, dataclasses._MISSING_TYPE) and getattr(self, field.name) is None: setattr(self, field.name, field.default) Adding this to your dataclass should then ensure that the default values are enforced without requiring a new default class.


Instruction:
Create random list of integers in Python

I'd like to create a random list of integers for testing purposes. The distribution of the numbers is not important. The only thing that is counting is time. I know generating random numbers is a time-consuming task, but there must be a better way. Here's my current solution: import random import timeit # Random lists from [0-999] interval print [random.randint(0, 1000) for r in xrange(10)] # v1 print [random.choice([i for i in xrange(1000)]) for r in xrange(10)] # v2 # Measurement: t1 = timeit.Timer('[random.randint(0, 1000) for r in xrange(10000)]', 'import random') # v1 t2 = timeit.Timer('random.sample(range(1000), 10000)', 'import random') # v2 print t1.timeit(1000)/1000 print t2.timeit(1000)/1000 v2 is faster than v1, but it is not working on such a large scale. It gives the following error: ValueError: sample larger than population Is there a fast, efficient solution that works at that scale? Some results from the answer Andrew's: 0.000290962934494 gnibbler's: 0.0058455221653 KennyTM's: 0.00219276118279 NumPy came, saw, and conquered.

Output: It is not entirely clear what you want, but I would use numpy.random.randint: import numpy.random as nprnd import timeit t1 = timeit.Timer('[random.randint(0, 1000) for r in xrange(10000)]', 'import random') # v1 ### Change v2 so that it picks numbers in (0, 10000) and thus runs... t2 = timeit.Timer('random.sample(range(10000), 10000)', 'import random') # v2 t3 = timeit.Timer('nprnd.randint(1000, size=10000)', 'import numpy.random as nprnd') # v3 print t1.timeit(1000)/1000 print t2.timeit(1000)/1000 print t3.timeit(1000)/1000 which gives on my machine: 0.0233682730198 0.00781716918945 0.000147947072983 Note that randint is very different from random.sample (in order for it to work in your case I had to change the 1,000 to 10,000 as one of the commentators pointed out -- if you really want them from 0 to 1,000 you could divide by 10). And if you really don't care what distribution you are getting then it is possible that you either don't understand your problem very well, or random numbers -- with apologies if that sounds rude...


Instruction:
Difference between pass statement and 3 dots(...) in python

What's the difference between the pass statement: def function(): pass and 3 dots: def function(): ... Which way is better and faster to execute(CPython)?

Output: pass has been in the language for a very long time and is just a no-op. It is designed to explicitly do nothing. ... is a token having the singleton value Ellipsis, similar to how None is a singleton value. Putting ... as your method body has the same effect as for example: def foo(): 1 The ... can be interpreted as a sentinel value where it makes sense from an API-design standpoint, e.g. if you overwrite __getitem__ to do something special if Ellipsis are passed, and then giving foo[...] special meaning. It is not specifically meant as a replacement for no-op stubs, though I have seen it being used that way and it doesn't hurt either


Instruction:
HDF5 - concurrency, compression & I/O performance

I have the following questions about HDF5 performance and concurrency: Does HDF5 support concurrent write access? Concurrency considerations aside, how is HDF5 performance in terms of I/O performance (does compression rates affect the performance)? Since I use HDF5 with Python, how does its performance compare to Sqlite? References: http://www.sqlite.org/faq.html#q5 Locking sqlite file on NFS filesystem possible? http://pandas.pydata.org/

Output: Updated to use pandas 0.13.1 No. http://pandas.pydata.org/pandas-docs/dev/io.html#notes-caveats. There are various ways to do this, e.g. have your different threads/processes write out the computation results, then have a single process combine. depending the type of data you store, how you do it, and how you want to retrieve, HDF5 can offer vastly better performance. Storing in an HDFStore as a single array, float data, compressed (in other words, not storing it in a format that allows for querying), will be stored/read amazingly fast. Even storing in the table format (which slows down the write performance), will offer quite good write performance. You can look at this for some detailed comparisons (which is what HDFStore uses under the hood). http://www.pytables.org/, here's a nice picture: Since PyTables 2.3 the queries are now indexed, so performance is actually MUCH better than this. To answer your question, if you want any kind of performance, HDF5 is the way to go. Writing: In [14]: %timeit test_sql_write(df) 1 loops, best of 3: 6.24 s per loop In [15]: %timeit test_hdf_fixed_write(df) 1 loops, best of 3: 237 ms per loop In [16]: %timeit test_hdf_table_write(df) 1 loops, best of 3: 901 ms per loop In [17]: %timeit test_csv_write(df) 1 loops, best of 3: 3.44 s per loop Reading In [18]: %timeit test_sql_read() 1 loops, best of 3: 766 ms per loop In [19]: %timeit test_hdf_fixed_read() 10 loops, best of 3: 19.1 ms per loop In [20]: %timeit test_hdf_table_read() 10 loops, best of 3: 39 ms per loop In [22]: %timeit test_csv_read() 1 loops, best of 3: 620 ms per loop And here's the code import sqlite3 import os from pandas.io import sql In [3]: df = DataFrame(randn(1000000,2),columns=list('AB')) <class 'pandas.core.frame.DataFrame'> Int64Index: 1000000 entries, 0 to 999999 Data columns (total 2 columns): A 1000000 non-null values B 1000000 non-null values dtypes: float64(2) def test_sql_write(df): if os.path.exists('test.sql'): os.remove('test.sql') sql_db = sqlite3.connect('test.sql') sql.write_frame(df, name='test_table', con=sql_db) sql_db.close() def test_sql_read(): sql_db = sqlite3.connect('test.sql') sql.read_frame("select * from test_table", sql_db) sql_db.close() def test_hdf_fixed_write(df): df.to_hdf('test_fixed.hdf','test',mode='w') def test_csv_read(): pd.read_csv('test.csv',index_col=0) def test_csv_write(df): df.to_csv('test.csv',mode='w') def test_hdf_fixed_read(): pd.read_hdf('test_fixed.hdf','test') def test_hdf_table_write(df): df.to_hdf('test_table.hdf','test',format='table',mode='w') def test_hdf_table_read(): pd.read_hdf('test_table.hdf','test') Of course YMMV.


Instruction:
How can I provide a "callback" to an API?

I was reading some module documentation and saw something I didn't understand, in the explanation of parameters for a method: callback - callback function which will be called with argument list equal to callbackargs+(result,) as soon as calculation is done callbackargs - additional arguments for callback function group - job group, is used when wait(group) is called to wait for jobs in a given group to finish How can I call the method and correctly supply arguments for these parameters? What are they used for, and how does this "callback" scheme work?

Output: A callback is a function provided by the consumer of an API that the API can then turn around and invoke (calling you back). If I setup a Dr.'s appointment, I can give them my phone number, so they can call me the day before to confirm the appointment. A callback is like that, except instead of just being a phone number, it can be arbitrary instructions like "send me an email at this address, and also call my secretary and have her put it in my calendar. Callbacks are often used in situations where an action is asynchronous. If you need to call a function, and immediately continue working, you can't sit there wait for its return value to let you know what happened, so you provide a callback. When the function is done completely its asynchronous work it will then invoke your callback with some predetermined arguments (usually some you supply, and some about the status and result of the asynchronous action you requested). If the Dr. is out of the office, or they are still working on the schedule, rather than having me wait on hold until he gets back, which could be several hours, we hang up, and once the appointment has been scheduled, they call me. In this specific case, the documented method will compute the result, put it together with any callbackargs specified, and call callback, passing it those values as the arguments.


Instruction:
Django download a file

I'm quite new to using Django and I am trying to develop a website where the user is able to upload a number of excel files, these files are then stored in a media folder Webproject/project/media. def upload(request): if request.POST: form = FileForm(request.POST, request.FILES) if form.is_valid(): form.save() return render_to_response('project/upload_successful.html') else: form = FileForm() args = {} args.update(csrf(request)) args['form'] = form return render_to_response('project/create.html', args) The document is then displayed in a list along with any other document they have uploaded, which you can click into and it will displays basic info about them and the name of the excelfile they have uploaded. From here I want to be able to download the same excel file again using the link: <a href="/project/download"> Download Document </a> My urls are urlpatterns = [ url(r'^$', ListView.as_view(queryset=Post.objects.all().order_by("-date")[:25], template_name="project/project.html")), url(r'^(?P<pk>\d+)$', DetailView.as_view(model=Post, template_name="project/post.html")), url(r'^upload/$', upload), url(r'^download/(?P<path>.*)$', serve, {'document root': settings.MEDIA_ROOT}), ] + static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT) but I get the error, serve() got an unexpected keyword argument 'document root'. can anyone explain how to fix this? OR Explain how I can get the uploaded files to to be selected and served using def download(request): file_name = #get the filename of desired excel file path_to_file = #get the path of desired excel file response = HttpResponse(mimetype='application/force-download') response['Content-Disposition'] = 'attachment; filename=%s' % smart_str(file_name) response['X-Sendfile'] = smart_str(path_to_file) return response

Output: You missed underscore in argument document_root. But it's bad idea to use serve in production. Use something like this instead: import os from django.conf import settings from django.http import HttpResponse, Http404 def download(request, path): file_path = os.path.join(settings.MEDIA_ROOT, path) if os.path.exists(file_path): with open(file_path, 'rb') as fh: response = HttpResponse(fh.read(), content_type="application/vnd.ms-excel") response['Content-Disposition'] = 'inline; filename=' + os.path.basename(file_path) return response raise Http404


Instruction:
Limit number of threads in numpy

It seems that my numpy library is using 4 threads, and setting OMP_NUM_THREADS=1 does not stop this. numpy.show_config() gives me these results: atlas_threads_info: libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/usr/lib64/atlas'] define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')] language = f77 include_dirs = ['/usr/include'] blas_opt_info: libraries = ['ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/usr/lib64/atlas'] define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')] language = c include_dirs = ['/usr/include'] atlas_blas_threads_info: libraries = ['ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/usr/lib64/atlas'] define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')] language = c include_dirs = ['/usr/include'] openblas_info: NOT AVAILABLE lapack_opt_info: libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas'] library_dirs = ['/usr/lib64/atlas'] define_macros = [('ATLAS_INFO', '"\\"3.8.4\\""')] language = f77 include_dirs = ['/usr/include'] So I know it is using blas, but I can't figure out how to make it use 1 thread for matrix multiplication.

Output: There are a few common multi CPU libraries that are used for numerical computations, including inside of NumPy. There are a few environment flags that you can set before running the script to limit the number of CPUS that they use. Try setting all of the following: export MKL_NUM_THREADS=1 export NUMEXPR_NUM_THREADS=1 export OMP_NUM_THREADS=1 Sometimes it's a bit tricky to see where exactly multithreading is introduced. Other answers show environment flags for other libraries. They may also work.


Instruction:
What is the purpose of classmethod in this code?

In django.utils.tree.py: def _new_instance(cls, children=None, connector=None, negated=False): obj = Node(children, connector, negated) obj.__class__ = cls return obj _new_instance = classmethod(_new_instance) I don't know what classmethod does in this code sample. Can someone explain what it does and how to use it?

Output: classmethod is a decorator, wrapping a function, and you can call the resulting object on a class or (equivalently) an instance thereof: >>> class x(object): ... def c1(*args): print 'c1', args ... c1 = classmethod(c1) ... @classmethod ... def c2(*args): print 'c2', args ... >>> inst = x() >>> x.c1() c1 (<class '__main__.x'>,) >>> x.c2() c2 (<class '__main__.x'>,) >>> inst.c1() c1 (<class '__main__.x'>,) >>> inst.c2() c2 (<class '__main__.x'>,) As you see, whether you define it directly or with decorator syntax, and whether you call it on the class or the instance, the classmethod always receives the class as its first argument. One of the main uses of classmethod is to define alternative constructors: >>> class y(object): ... def __init__(self, astring): ... self.s = astring ... @classmethod ... def fromlist(cls, alist): ... x = cls('') ... x.s = ','.join(str(s) for s in alist) ... return x ... def __repr__(self): ... return 'y(%r)' % self.s ... >>> y1 = y('xx') >>> y1 y('xx') >>> y2 = y.fromlist(range(3)) >>> y2 y('0,1,2') Now if you subclass y, the classmethod keeps working, e.g.: >>> class k(y): ... def __repr__(self): ... return 'k(%r)' % self.s.upper() ... >>> k1 = k.fromlist(['za','bu']) >>> k1 k('ZA,BU')


Instruction:
Sqlalchemy if table does not exist

I wrote a module which is to create an empty database file def create_database(): engine = create_engine("sqlite:///myexample.db", echo=True) metadata = MetaData(engine) metadata.create_all() But in another function, I want to open myexample.db database, and create tables to it if it doesn't already have that table. EG of the first, subsequent table I would create would be: Table(Variable_TableName, metadata, Column('Id', Integer, primary_key=True, nullable=False), Column('Date', Date), Column('Volume', Float)) (Since it is initially an empty database, it will have no tables in it, but subsequently, I can add more tables to it. Thats what i'm trying to say.) Any suggestions?

Output: I've managed to figure out what I intended to do. I used engine.dialect.has_table(engine, Variable_tableName) to check if the database has the table inside. IF it doesn't, then it will proceed to create a table in the database. Sample code: engine = create_engine("sqlite:///myexample.db") # Access the DB Engine if not engine.dialect.has_table(engine, Variable_tableName): # If table don't exist, Create. metadata = MetaData(engine) # Create a table with the appropriate Columns Table(Variable_tableName, metadata, Column('Id', Integer, primary_key=True, nullable=False), Column('Date', Date), Column('Country', String), Column('Brand', String), Column('Price', Float), # Implement the creation metadata.create_all() This seems to be giving me what i'm looking for.