Instruction:
get UTC timestamp in python with datetime

Is there a way to get the UTC timestamp by specifying the date? What I would expect: datetime(2008, 1, 1, 0, 0, 0, 0) should result in 1199145600 Creating a naive datetime object means that there is no time zone information. If I look at the documentation for datetime.utcfromtimestamp, creating a UTC timestamp means leaving out the time zone information. So I would guess, that creating a naive datetime object (like I did) would result in a UTC timestamp. However: then = datetime(2008, 1, 1, 0, 0, 0, 0) datetime.utcfromtimestamp(float(then.strftime('%s'))) results in 2007-12-31 23:00:00 Is there still any hidden time zone information in the datetime object? What am I doing wrong?

Output: Naïve datetime versus aware datetime Default datetime objects are said to be "naïve": they keep time information without the time zone information. Think about naïve datetime as a relative number (ie: +4) without a clear origin (in fact your origin will be common throughout your system boundary). In contrast, think about aware datetime as absolute numbers (ie: 8) with a common origin for the whole world. Without timezone information you cannot convert the "naive" datetime towards any non-naive time representation (where does +4 targets if we don't know from where to start ?). This is why you can't have a datetime.datetime.toutctimestamp() method. (cf: http://bugs.python.org/issue1457227) To check if your datetime dt is naïve, check dt.tzinfo, if None, then it's naïve: datetime.now() ## DANGER: returns naïve datetime pointing on local time datetime(1970, 1, 1) ## returns naïve datetime pointing on user given time I have naïve datetimes, what can I do ? You must make an assumption depending on your particular context: The question you must ask yourself is: was your datetime on UTC ? or was it local time ? If you were using UTC (you are out of trouble): import calendar def dt2ts(dt): """Converts a datetime object to UTC timestamp naive datetime will be considered UTC. """ return calendar.timegm(dt.utctimetuple()) If you were NOT using UTC, welcome to hell. You have to make your datetime non-naïve prior to using the former function, by giving them back their intended timezone. You'll need the name of the timezone and the information about if DST was in effect when producing the target naïve datetime (the last info about DST is required for cornercases): import pytz ## pip install pytz mytz = pytz.timezone('Europe/Amsterdam') ## Set your timezone dt = mytz.normalize(mytz.localize(dt, is_dst=True)) ## Set is_dst accordingly Consequences of not providing is_dst: Not using is_dst will generate incorrect time (and UTC timestamp) if target datetime was produced while a backward DST was put in place (for instance changing DST time by removing one hour). Providing incorrect is_dst will of course generate incorrect time (and UTC timestamp) only on DST overlap or holes. And, when providing also incorrect time, occuring in "holes" (time that never existed due to forward shifting DST), is_dst will give an interpretation of how to consider this bogus time, and this is the only case where .normalize(..) will actually do something here, as it'll then translate it as an actual valid time (changing the datetime AND the DST object if required). Note that .normalize() is not required for having a correct UTC timestamp at the end, but is probably recommended if you dislike the idea of having bogus times in your variables, especially if you re-use this variable elsewhere. and AVOID USING THE FOLLOWING: (cf: Datetime Timezone conversion using pytz) dt = dt.replace(tzinfo=timezone('Europe/Amsterdam')) ## BAD !! Why? because .replace() replaces blindly the tzinfo without taking into account the target time and will choose a bad DST object. Whereas .localize() uses the target time and your is_dst hint to select the right DST object. OLD incorrect answer (thanks @J.F.Sebastien for bringing this up): Hopefully, it is quite easy to guess the timezone (your local origin) when you create your naive datetime object as it is related to the system configuration that you would hopefully NOT change between the naive datetime object creation and the moment when you want to get the UTC timestamp. This trick can be used to give an imperfect question. By using time.mktime we can create an utc_mktime: def utc_mktime(utc_tuple): """Returns number of seconds elapsed since epoch Note that no timezone are taken into consideration. utc tuple must be: (year, month, day, hour, minute, second) """ if len(utc_tuple) == 6: utc_tuple += (0, 0, 0) return time.mktime(utc_tuple) - time.mktime((1970, 1, 1, 0, 0, 0, 0, 0, 0)) def datetime_to_timestamp(dt): """Converts a datetime object to UTC timestamp""" return int(utc_mktime(dt.timetuple())) You must make sure that your datetime object is created on the same timezone than the one that has created your datetime. This last solution is incorrect because it makes the assumption that the UTC offset from now is the same than the UTC offset from EPOCH. Which is not the case for a lot of timezones (in specific moment of the year for the Daylight Saving Time (DST) offsets).


Instruction:
memory-efficient built-in SqlAlchemy iterator/generator?

I have a ~10M record MySQL table that I interface with using SqlAlchemy. I have found that queries on large subsets of this table will consume too much memory even though I thought I was using a built-in generator that intelligently fetched bite-sized chunks of the dataset: for thing in session.query(Things): analyze(thing) To avoid this, I find I have to build my own iterator that bites off in chunks: lastThingID = None while True: things = query.filter(Thing.id < lastThingID).limit(querySize).all() if not rows or len(rows) == 0: break for thing in things: lastThingID = row.id analyze(thing) Is this normal or is there something I'm missing regarding SA built-in generators? The answer to this question seems to indicate that the memory consumption is not to be expected.

Output: Most DBAPI implementations fully buffer rows as they are fetched - so usually, before the SQLAlchemy ORM even gets a hold of one result, the whole result set is in memory. But then, the way Query works is that it fully loads the given result set by default before returning to you your objects. The rationale here regards queries that are more than simple SELECT statements. For example, in joins to other tables that may return the same object identity multiple times in one result set (common with eager loading), the full set of rows needs to be in memory so that the correct results can be returned otherwise collections and such might be only partially populated. So Query offers an option to change this behavior through yield_per(). This call will cause the Query to yield rows in batches, where you give it the batch size. As the docs state, this is only appropriate if you aren't doing any kind of eager loading of collections so it's basically if you really know what you're doing. Also, if the underlying DBAPI pre-buffers rows, there will still be that memory overhead so the approach only scales slightly better than not using it. I hardly ever use yield_per(); instead, I use a better version of the LIMIT approach you suggest above using window functions. LIMIT and OFFSET have a huge problem that very large OFFSET values cause the query to get slower and slower, as an OFFSET of N causes it to page through N rows - it's like doing the same query fifty times instead of one, each time reading a larger and larger number of rows. With a window-function approach, I pre-fetch a set of "window" values that refer to chunks of the table I want to select. I then emit individual SELECT statements that each pull from one of those windows at a time. The window function approach is on the wiki and I use it with great success. Also note: not all databases support window functions; you need Postgresql, Oracle, or SQL Server. IMHO using at least Postgresql is definitely worth it - if you're using a relational database, you might as well use the best.


Instruction:
Removing the first folder in a path

I have a path which looks like /First/Second/Third/Fourth/Fifth and I would like to remove the First from it, thus obtaining Second/Third/Fourth/Fifth The only idea I could come up with is to use recursively os.path.split but this does not seem optimal. Is there a better solution?

Output: There really is nothing in the os.path module to do this. Every so often, someone suggests creating a splitall function that returns a list (or iterator) of all of the components, but it never gained enough traction. Partly this is because every time anyone ever suggested adding new functionality to os.path, it re-ignited the long-standing dissatisfaction with the general design of the library, leading to someone proposing a new, more OO-like, API for paths to deprecated the os, clunky API. In 3.4, that finally happened, with pathlib. And it's already got functionality that wasn't in os.path. So: >>> import pathlib >>> p = pathlib.Path('/First/Second/Third/Fourth/Fifth') >>> p.parts[2:] ('Second', 'Third', 'Fourth', 'Fifth') >>> pathlib.Path(*p.parts[2:]) PosixPath('Second/Third/Fourth/Fifth') Or… are you sure you really want to remove the first component, rather than do this? >>> p.relative_to(*p.parts[:2]) PosixPath('Second/Third/Fourth/Fifth') If you need to do this in 2.6-2.7 or 3.2-3.3, there's a backport of pathlib. Of course, you can use string manipulation, as long as you're careful to normalize the path and use os.path.sep, and to make sure you handle the fiddly details with non-absolute paths or with systems with drive letters, and… Or you can just wrap up your recursive os.path.split. What exactly is "non-optimal" about it, once you wrap it up? It may be a bit slower, but we're talking nanoseconds here, many orders of magnitude faster than even calling stat on a file. It will have recursion-depth problems if you have a filesystem that's 1000 directories deep, but have you ever seen one? (If so, you can always turn it into a loop…) It takes a few minutes to wrap it up and write good unit tests, but that's something you just do once and never worry about again. So, honestly, if you don't want to use pathlib, that's what I'd do.


Instruction:
How do I make an auto increment integer field in Django?

I am making an Order model for a shopping cart and I need to make a field that auto increments when the order is made: class Order(models.Model): cart = models.ForeignKey(Cart) add_date = models.DateTimeField(auto_now_add=True) order_number = models.IntegerField() enable = models.BooleanField(default=True) How do I make the IntegerField auto increment?

Output: In Django 1 : Django model class has default field with name id which is auto increment Field. 2 : You can define your own auto increment field using AutoField field. class Order(models.Model): auto_increment_id = models.AutoField(primary_key=True) # primary_key = True if you do not want to use default field "id" given by django to your model db design +------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Table | Create Table | +------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+ | core_order | CREATE TABLE `core_order` ( `auto_increment_id` int(11) NOT NULL AUTO_INCREMENT, PRIMARY KEY (`auto_increment_id`) ) ENGINE=InnoDB DEFAULT CHARSET=latin1 | +------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+ 1 row in set (0.01 sec) If you want to use django's default id as increment field . class Order(models.Model): add_date = models.DateTimeField(auto_now_add=True) db design +-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Table | Create Table | +-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+ | core_order | CREATE TABLE `core_order` ( `id` int(11) NOT NULL AUTO_INCREMENT, `add_date` datetime NOT NULL, PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=latin1 | +-------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+


Instruction:
Convert string into datetime.time object

Given the string in this format "HH:MM", for example "03:55", that represents 3 hours and 55 minutes. I want to convert it to datetime.time object for easier manipulation. What would be the easiest way to do that?

Output: Use datetime.datetime.strptime() and call .time() on the result: >>> datetime.datetime.strptime('03:55', '%H:%M').time() datetime.time(3, 55) The first argument to .strptime() is the string to parse, the second is the expected format.


Instruction:
Is it possible to insert a row at an arbitrary position in a dataframe using pandas?

I have a DataFrame object similar to this one: onset length 1 2.215 1.3 2 23.107 1.3 3 41.815 1.3 4 61.606 1.3 ... What I would like to do is insert a row at a position specified by some index value and update the following indices accordingly. E.g.: onset length 1 2.215 1.3 2 23.107 1.3 3 30.000 1.3 # new row 4 41.815 1.3 5 61.606 1.3 ... What would be the best way to do this?

Output: You could slice and use concat to get what you want. from pandas import DataFrame, concat line = DataFrame({"onset": 30.0, "length": 1.3}, index=[3]) df2 = concat([df.iloc[:2], line, df.iloc[2:]]).reset_index(drop=True) This will produce the dataframe in your example output. As far as I'm aware, concat is the best method to achieve an insert type operation in pandas, but admittedly I'm by no means a pandas expert.


Instruction:
How to use inspect to get the caller's info from callee in Python?

I need to get the caller info (what file/what line) from the callee. I learned that I can use the inspect module for that purpose, but not exactly how. How do you get that info with inspection? Or is there any other way to get the info? import inspect print __file__ c=inspect.currentframe() print c.f_lineno def hello(): print inspect.stack ?? what file called me in what line? hello()

Output: The caller's frame is one frame higher than the current frame. You can use inspect.currentframe().f_back to find the caller's frame. Then use inspect.getframeinfo to get the caller's filename and line number. import inspect def hello(): previous_frame = inspect.currentframe().f_back ( filename, line_number, function_name, lines, index, ) = inspect.getframeinfo(previous_frame) return (filename, line_number, function_name, lines, index) print(hello()) # ('/home/unutbu/pybin/test.py', 10, '<module>', ['hello()\n'], 0)


Instruction:
Sum the digits of a number

If I want to find the sum of the digits of a number, i.e.: Input: 932 Output: 14, which is (9 + 3 + 2) What is the fastest way of doing this? I instinctively did: sum(int(digit) for digit in str(number)) and I found this online: sum(map(int, str(number))) Which is best to use for speed, and are there any other methods which are even faster?

Output: Both lines you posted are fine, but you can do it purely in integers, and it will be the most efficient: def sum_digits(n): s = 0 while n: s += n % 10 n //= 10 return s or with divmod: def sum_digits2(n): s = 0 while n: n, remainder = divmod(n, 10) s += remainder return s Slightly faster is using a single assignment statement: def sum_digits3(n): r = 0 while n: r, n = r + n % 10, n // 10 return r > %timeit sum_digits(n) 1000000 loops, best of 3: 574 ns per loop > %timeit sum_digits2(n) 1000000 loops, best of 3: 716 ns per loop > %timeit sum_digits3(n) 1000000 loops, best of 3: 479 ns per loop > %timeit sum(map(int, str(n))) 1000000 loops, best of 3: 1.42 us per loop > %timeit sum([int(digit) for digit in str(n)]) 100000 loops, best of 3: 1.52 us per loop > %timeit sum(int(digit) for digit in str(n)) 100000 loops, best of 3: 2.04 us per loop


Instruction:
Get the nth item of a generator in Python

Is there a more syntactically concise way of writing the following? gen = (i for i in xrange(10)) index = 5 for i, v in enumerate(gen): if i is index: return v It seems almost natural that a generator should have a gen[index] expression, that acts as a list, but is functionally identical to the above code.

Output: one method would be to use itertools.islice >>> gen = (x for x in range(10)) >>> index = 5 >>> next(itertools.islice(gen, index, None)) 5


Instruction:
How do I install a library permanently in Colab?

In Google Colaboratory, I can install a new library using !pip install package-name. But when I open the notebook again tomorrow, I need to re-install it every time. Is there a way to install a library permanently? No need to spend time installing every time to use?


Instruction:
How do I read and write CSV files?

How do I read the following CSV file? 1,"A towel,",1.0 42," it says, ",2.0 1337,is about the most ,-1 0,massively useful thing ,123 -2,an interstellar hitchhiker can have.,3 How do I write the following data to a CSV file? data = [ (1, "A towel,", 1.0), (42, " it says, ", 2.0), (1337, "is about the most ", -1), (0, "massively useful thing ", 123), (-2, "an interstellar hitchhiker can have.", 3), ]

Output: Here are some minimal complete examples how to read CSV files and how to write CSV files with Python. Pure Python: import csv # Define data data = [ (1, "A towel,", 1.0), (42, " it says, ", 2.0), (1337, "is about the most ", -1), (0, "massively useful thing ", 123), (-2, "an interstellar hitchhiker can have.", 3), ] # Write CSV file with open("test.csv", "wt") as fp: writer = csv.writer(fp, delimiter=",") # writer.writerow(["your", "header", "foo"]) # write header writer.writerows(data) # Read CSV file with open("test.csv") as fp: reader = csv.reader(fp, delimiter=",", quotechar='"') # next(reader, None) # skip the headers data_read = [row for row in reader] print(data_read) After that, the contents of data_read are [['1', 'A towel,', '1.0'], ['42', ' it says, ', '2.0'], ['1337', 'is about the most ', '-1'], ['0', 'massively useful thing ', '123'], ['-2', 'an interstellar hitchhiker can have.', '3']] Please note that CSV reads only strings. You need to convert to the column types manually. A Python 2+3 version was here before (link), but Python 2 support is dropped. Removing the Python 2 stuff massively simplified this answer. Related How do I write data into csv format as string (not file)? How can I use io.StringIO() with the csv module?: This is interesting if you want to serve a CSV on-the-fly with Flask, without actually storing the CSV on the server. mpu Have a look at my utility package mpu for a super simple and easy to remember one: import mpu.io data = mpu.io.read('example.csv', delimiter=',', quotechar='"', skiprows=None) mpu.io.write('example.csv', data) Pandas import pandas as pd # Read the CSV into a pandas data frame (df) # With a df you can do many things # most important: visualize data with Seaborn df = pd.read_csv('myfile.csv', sep=',') print(df) # Or export it in many ways, e.g. a list of tuples tuples = [tuple(x) for x in df.values] # or export it as a list of dicts dicts = df.to_dict().values() See read_csv docs for more information. Please note that pandas automatically infers if there is a header line, but you can set it manually, too. If you haven't heard of Seaborn, I recommend having a look at it. Other Reading CSV files is supported by a bunch of other libraries, for example: dask.dataframe.read_csv spark.read.csv Created CSV file 1,"A towel,",1.0 42," it says, ",2.0 1337,is about the most ,-1 0,massively useful thing ,123 -2,an interstellar hitchhiker can have.,3 Common file endings .csv Working with the data After reading the CSV file to a list of tuples / dicts or a Pandas dataframe, it is simply working with this kind of data. Nothing CSV specific. Alternatives JSON: Nice for writing human-readable data; VERY commonly used (read & write) CSV: Super simple format (read & write) YAML: Nice to read, similar to JSON (read & write) pickle: A Python serialization format (read & write) MessagePack (Python package): More compact representation (read & write) HDF5 (Python package): Nice for matrices (read & write) XML: exists too *sigh* (read & write) For your application, the following might be important: Support by other programming languages Reading / writing performance Compactness (file size) See also: Comparison of data serialization formats In case you are rather looking for a way to make configuration files, you might want to read my short article Configuration files in Python


Instruction:
namedtuple vs NamedTuple in Python

The typing module documentation says that the two code snippets below are equivalent: from typing import NamedTuple class Employee(NamedTuple): name: str id: int and: from collections import namedtuple Employee = namedtuple('Employee', ['name', 'id']) Are they the exact same thing or, if not, what are the differences between the two implementations?

Output: The type generated by subclassing typing.NamedTuple is equivalent to a collections.namedtuple, but with __annotations__, _field_types and _field_defaults attributes added. The generated code will behave the same, for all practical purposes, since nothing in Python currently acts on those typing related attributes (your IDE might use them, though). As a developer, using the typing module for your namedtuples allows a more natural declarative interface: You can easily specify default values for the fields (edit: in Python 3.7, collections.namedtuple got a new defaults keyword so this is no longer an advantage) You don't need to repeat the type name twice ("Employee") You can customize the type directly (e.g. adding a docstring or some methods) As before, your class will be a subclass of tuple, and instances will be instances of tuple as usual. Interestingly, your class will not be a subclass of NamedTuple. If you want to know why, read on for more info about the implementation detail. from typing import NamedTuple class Employee(NamedTuple): name: str id: int Behaviour in Python <= 3.8 >>> issubclass(Employee, NamedTuple) False >>> isinstance(Employee(name='guido', id=1), NamedTuple) False typing.NamedTuple is a class, it uses metaclasses and a custom __new__ to handle the annotations, and then it delegates to collections.namedtuple to build and return the type. As you may have guessed from the lowercased name convention, collections.namedtuple is not a type/class - it's a factory function. It works by building up a string of Python source code, and then calling exec on this string. The generated constructor is plucked out of a namespace and included in a 3-argument invocation of the metaclass type to build and return your class. This explains the weird inheritance breakage seen above, NamedTuple uses a metaclass in order to use a different metaclass to instantiate the class object. Behaviour in Python >= 3.9 typing.NamedTuple is changed from a type (class) to a function (def) >>> issubclass(Employee, NamedTuple) TypeError: issubclass() arg 2 must be a class or tuple of classes >>> isinstance(Employee(name="guido", id=1), NamedTuple) TypeError: isinstance() arg 2 must be a type or tuple of types The metaclass acrobatics are gone, now it's just a simple factory function which calls collections.namedtuple and then sets __annotations__ on the returned type. Multiple inheritance using NamedTuple is now disallowed (it did not work properly in the first place). See bpo40185 / GH-19371 for the change.


Instruction:
Difference between entry_points/console_scripts and scripts in setup.py?

There are basically two ways to install Python console scripts to my path by setup.py: setup( ... entry_points = { 'console_scripts': [ 'foo = package.module:func', ], } ) and setup( ... scripts = [ 'scripts/myscript.sh' ] ) What are the differences? I see the first approach allows me to choose nice, specific name for my script, but are there any other differences? Different original purposes, compatibility (setuptools, distutils, ...?), usage, ...? I am quite confused and a nice elaborated reply could help me (and probably also others) to properly understand all this. Update: Since I asked the question PyPA published these cool docs on the topic.

Output: The docs for the (awesome) Click package suggest a few reasons to use entry points instead of scripts, including cross-platform compatibility and avoiding having the interpreter assign __name__ to __main__, which could cause code to be imported twice (if another module imports your script) Click is a nice way to implement functions for use as entry_points, btw.


Instruction:
Convert dictionary entries into variables

Is there a Pythonic way to assign the values of a dictionary to its keys, in order to convert the dictionary entries into variables? I tried this out: >>> d = {'a':1, 'b':2} >>> for key,val in d.items(): exec('exec(key)=val') exec(key)=val ^ SyntaxError: invalid syntax I am certain that the key-value pairs are correct because they were previously defined as variables by me before. I then stored these variables in a dictionary (as key-value pairs) and would like to reuse them in a different function. I could just define them all over again in the new function, but because I may have a dictionary with about 20 entries, I thought there may be a more efficient way of doing this.

Output: This was what I was looking for: d = {'a':1, 'b':2} for key,val in d.items(): exec(key + '=val') NOTE: As noted by @divenex in the comments, this solution only creates global variables -- it will not create local variables in a function. Move the code inside a function and you will get an error. def func(): d = {'a':1, 'b':2} for key,val in d.items(): exec(key + '=val') print(a,b) func() Error message: NameError: name 'a' is not defined


Instruction:
How to change the Spyder editor background to dark?

I've just updated Spyder to version 3.1 and I'm having trouble changing the colour scheme to dark. I've been able to change the Python and iPython console's to dark but the option to change the editor to dark is not where I would expect it to be. Could anybody tell me how to change the colour scheme of the Spyder 3.1 editor to dark?

Output: If you're using Spyder 3, please go to Tools > Preferences > Syntax Coloring and select there the dark theme you want to use. In Spyder 4, a dark theme is used by default. But if you want to select a different theme you can go to Tools > Preferences > Appearance > Syntax highlighting theme


Instruction:
Interactive large plot with ~20 million sample points and gigabytes of data

I have got a problem (with my RAM) here: it's not able to hold the data I want to plot. I do have sufficient HD space. Is there any solution to avoid that "shadowing" of my data-set? Concretely I deal with Digital Signal Processing and I have to use a high sample-rate. My framework (GNU Radio) saves the values (to avoid using too much disk space) in binary. I unpack it. Afterwards I need to plot. I need the plot zoomable, and interactive. And that is an issue. Is there any optimization potential to this, or another software/programming language (like R or so) which can handle larger data-sets? Actually I want much more data in my plots. But I have no experience with other software. GNUplot fails, with a similar approach to the following. I don't know R (jet). import matplotlib.pyplot as plt import matplotlib.cbook as cbook import struct """ plots a cfile cfile - IEEE single-precision (4-byte) floats, IQ pairs, binary txt - index,in-phase,quadrature in plaintext note: directly plotting with numpy results into shadowed functions """ # unpacking the cfile dataset def unpack_set(input_filename, output_filename): index = 0 # index of the samples output_filename = open(output_filename, 'wb') with open(input_filename, "rb") as f: byte = f.read(4) # read 1. column of the vector while byte != "": # stored Bit Values floati = struct.unpack('f', byte) # write value of 1. column to a variable byte = f.read(4) # read 2. column of the vector floatq = struct.unpack('f', byte) # write value of 2. column to a variable byte = f.read(4) # next row of the vector and read 1. column # delimeter format for matplotlib lines = ["%d," % index, format(floati), ",", format(floatq), "\n"] output_filename.writelines(lines) index = index + 1 output_filename.close return output_filename.name # reformats output (precision configuration here) def format(value): return "%.8f" % value # start def main(): # specify path unpacked_file = unpack_set("test01.cfile", "test01.txt") # pass file reference to matplotlib fname = str(unpacked_file) plt.plotfile(fname, cols=(0,1)) # index vs. in-phase # optional # plt.axes([0, 0.5, 0, 100000]) # for 100k samples plt.grid(True) plt.title("Signal-Diagram") plt.xlabel("Sample") plt.ylabel("In-Phase") plt.show(); if __name__ == "__main__": main() Something like plt.swap_on_disk() could cache the stuff on my SSD ;)

Output: So your data isn't that big, and the fact that you're having trouble plotting it points to issues with the tools. Matplotlib has lots of options and the output is fine, but it's a huge memory hog and it fundamentally assumes your data is small. But there are other options out there. So as an example, I generated a 20M data-point file 'bigdata.bin' using the following: #!/usr/bin/env python import numpy import scipy.io.numpyio npts=20000000 filename='bigdata.bin' def main(): data = (numpy.random.uniform(0,1,(npts,3))).astype(numpy.float32) data[:,2] = 0.1*data[:,2]+numpy.exp(-((data[:,1]-0.5)**2.)/(0.25**2)) fd = open(filename,'wb') scipy.io.numpyio.fwrite(fd,data.size,data) fd.close() if __name__ == "__main__": main() This generates a file of size ~229MB, which isn't all that big; but you've expressed that you'd like to go to even larger files, so you'll hit memory limits eventually. Let's concentrate on non-interactive plots first. The first thing to realize is that vector plots with glyphs at each point are going to be a disaster -- for each of the 20 M points, most of which are going to overlap anyway, trying to render little crosses or circles or something is going to be a diaster, generating huge files and taking tonnes of time. This, I think is what is sinking matplotlib by default. Gnuplot has no trouble dealing with this: gnuplot> set term png gnuplot> set output 'foo.png' gnuplot> plot 'bigdata.bin' binary format="%3float32" using 2:3 with dots And even Matplotlib can be made to behave with some caution (choosing a raster back end, and using pixels to mark points): #!/usr/bin/env python import numpy import matplotlib matplotlib.use('Agg') import matplotlib.pyplot as plt datatype=[('index',numpy.float32), ('floati',numpy.float32), ('floatq',numpy.float32)] filename='bigdata.bin' def main(): data = numpy.memmap(filename, datatype, 'r') plt.plot(data['floati'],data['floatq'],'r,') plt.grid(True) plt.title("Signal-Diagram") plt.xlabel("Sample") plt.ylabel("In-Phase") plt.savefig('foo2.png') if __name__ == "__main__": main() Now, if you want interactive, you're going to have to bin the data to plot, and zoom in on the fly. I don't know of any python tools that will help you do this offhand. On the other hand, plotting-big-data is a pretty common task, and there are tools that are up for the job. Paraview is my personal favourite, and VisIt is another one. They both are mainly for 3D data, but Paraview in particular does 2d as well, and is very interactive (and even has a Python scripting interface). The only trick will be to write the data into a file format that Paraview can easily read.


Instruction:
Executing an SQL query on a Pandas dataset

I have a Pandas dataset called df. How can I do: df.query("select * from df")

Output: This is not what pandas.query is supposed to do. You can look at package pandasql (same like sqldf in R ) Update: Note pandasql hasn't been maintained since 2017. Use another library from an answer below. import pandas as pd import pandasql as ps df = pd.DataFrame([[1234, 'Customer A', '123 Street', np.nan], [1234, 'Customer A', np.nan, '333 Street'], [1233, 'Customer B', '444 Street', '333 Street'], [1233, 'Customer B', '444 Street', '666 Street']], columns= ['ID', 'Customer', 'Billing Address', 'Shipping Address']) q1 = """SELECT ID FROM df """ print(ps.sqldf(q1, locals())) ID 0 1234 1 1234 2 1233 3 1233 Update 2020-07-10 update the pandasql ps.sqldf("select * from df")


Instruction:
How can I login to a website with Python?

How can I do it? I was trying to enter some specified link (with urllib), but to do it, I need to log in. I have this source from the site: <form id="login-form" action="auth/login" method="post">
  Email <input id="email-email" type="text" name="handle" value="" autocomplete="off" /> Combination <input id="password-clear" type="text" value="Combination" autocomplete="off" /> <input id="password-password" type="password" name="password" value="" autocomplete="off" /> <input id="sumbitLogin" class="signin" type="submit" value="Sign In" /> Is this possible?

Output: Maybe you want to use twill. It's quite easy to use and should be able to do what you want. It will look like the following: from twill.commands import * go('http://example.org') fv("1", "email-email", "blabla.com") fv("1", "password-clear", "testpass") submit('0') You can use showforms() to list all forms once you used go… to browse to the site you want to login. Just try it from the python interpreter.


Instruction:
How to disable welcome message when importing pygame

When I import pygame, it prints the version and welcome message. The message reads: pygame 1.9.4 Hello from the pygame community. https://www.pygame.org/contribute.html Why is this printed? How can I disable this message?

Output: As can be seen in the source code, the message is not printed if the environment variable PYGAME_HIDE_SUPPORT_PROMPT is set. So the following code could be used to import pygame without printing the message: import os os.environ['PYGAME_HIDE_SUPPORT_PROMPT'] = "hide" import pygame Note that the value does not have to be "hide" but can be anything else as well, and the environment variable can also be set in other ways to achieve the same.


Instruction:
python requests get cookies

x = requests.post(url, data=data) print x.cookies I used the requests library to get some cookies from a website, but I can only get the cookies from the Response, how to get the cookies from the Request? Thanks!

Output: Alternatively, you can use requests.Session and observe cookies before and after a request: >>> import requests >>> session = requests.Session() >>> print(session.cookies.get_dict()) {} >>> response = session.get('http://google.com') >>> print(session.cookies.get_dict()) {'PREF': 'ID=5514c728c9215a9a:FF=0:TM=1406958091:LM=1406958091:S=KfAG0U9jYhrB0XNf', 'NID': '67=TVMYiq2wLMNvJi5SiaONeIQVNqxSc2RAwVrCnuYgTQYAHIZAGESHHPL0xsyM9EMpluLDQgaj3db_V37NjvshV-eoQdA8u43M8UwHMqZdL-S2gjho8j0-Fe1XuH5wYr9v'}


Instruction:
Can Golang multiply strings like Python can?

Python can multiply strings like so: Python 3.4.3 (default, Mar 26 2015, 22:03:40) [GCC 4.9.2] on linux Type "help", "copyright", "credits" or "license" for more information. >>> x = 'my new text is this long' >>> y = '#' * len(x) >>> y '########################' >>> Can Golang do the equivalent somehow?

Output: It has a function instead of an operator, strings.Repeat. Here's a port of your Python example, which you can run here: package main import ( "fmt" "strings" "unicode/utf8" ) func main() { x := "my new text is this long" y := strings.Repeat("#", utf8.RuneCountInString(x)) fmt.Println(x) fmt.Println(y) } Note that I've used utf8.RuneCountInString(x) instead of len(x); the former counts "runes" (Unicode code points), while the latter, when called on a string, counts bytes. In the case of "my new text is this long", the difference doesn't matter since all the runes are only one byte each, but it's good to get into the habit of specifying what you mean: len("ā") //=> 2 utf8.RuneCountInString("ā") //=> 1 An alternative to calling RuneCountInString is to convert the string to an array of runes and then call len on that: y := strings.Repeat("#", len([]rune(x))) But if all you're doing with the runes is counting them, I think it's clearer to use the utf8 function. Since this was a Python comparison question, note that the Python version of len also counts different things depending on what you call it on. In Python 2, it counted bytes on plain strings and runes on Unicode strings (u'...'): Python 2.7.18 (default, Sep 10 2022, 16:30:21) >>> len('ā') #=> 2 >>> len(u'ā') #=> 1 Whereas in modern Python, plain strings are Unicode strings; if you want to count bytes, you need to encode the string into a bytearray first: Python 3.12.0 (main, Oct 13 2023, 15:35:30) >>> len('ā') #=> 1 >>> len('ā'.encode('UTF-8')) #=> 2 So Python has multiple types of string; Go has only one kind of string, but different ways of dealing with its contents. Oh, it's also worth noting that the Golang concept of a "rune" doesn't (and can't) solve the problem that in Unicode, the question "How much string is one character?" does not always have a well-defined answer. I used "ā" above as an example of a string that's two bytes long containing only one rune (specifically U+0101 LATIN SMALL LETTER A WITH MACRON). But you could get what looks like that same string ("ā") by instead combining two runes (U+0061 LATIN SMALL LETTER A and U+0304 COMBINING MACRON), giving rune count 2 and byte len() 4. Proper Unicode processing will treat both forms as equal to each other (and convert one to the other depending on which Normalization Form is selected) but there's not really any sense in which the Platonic ideal string they're both equivalent to can be said to contain a definite number of runes.


Instruction:
How to check if particular value (in cell) is NaN in pandas DataFrame?

Let's say I have the following pandas DataFrame: import pandas as pd import numpy as np df = pd.DataFrame({"A": [1, np.nan, 2], "B": [5, 6, 0]}) Which would look like: >>> df A B 0 1.0 5 1 NaN 6 2 2.0 0 First option I know one way to check if a particular value is NaN: >>> df.isnull().iloc[1,0] True But this checks the whole dataframe just to get one value, so I imagine it's wasteful. Second option (not working) I thought below option, using iloc, would work as well, but it doesn't: >>> df.iloc[1,0] == np.nan False However if I check that value I get: >>> df.iloc[1,0] nan So, why is the second option not working? Is it possible to check for NaN values using iloc? Editor's note: This question previously used pd.np instead of np and .ix in addition to .iloc, but since these no longer exist, they have been edited out to keep it short and clear.

Output: Try pd.isna(): In [7]: pd.isna(df.iloc[1,0]) Out[7]: True AKA pd.isnull


Instruction:
Cloning private github repository within organisation in actions

I have 2 private GitHub repositories (say A and B) in the organization (say ORG). Repository A has repository B in requirements.txt: -e git+git@github.com:ORG/B.git#egg=B And I have the following workflow for A (in .github/workflows/test.yml): name: Python package on: push jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v1 - name: Install requirements run: | pip install -r requirements.txt - name: Test with pytest run: | pytest ./tests As B is private, it fails on installing it. Is it possible to install B while testing A in this workflow if they are in the same organization? How?

Output: Since access tokens are bound to an account and have write access to all its private repos, it's a very bad solution. Instead, use deploy keys. Deploy keys are simply SSH keys that you can use to clone a repo. Create a new SSH key pair on your computer Put the public key in the private dependency repo's Deploy keys Put the private key in the app repo's Actions secrets Delete the keys from your computer Once it's set, you can set the private key in the GitHub Action's SSH Agent. There's no need to import a third-party GitHub Action, a 2-liner will suffice. eval `ssh-agent -s` ssh-add - <<< '${{ secrets.PRIVATE_SSH_KEY }}' pip install -r requirements.txt I found that ssh-add command here.


Instruction:
Define a method outside of class definition?

class MyClass: def myFunc(self): pass Can I create MyFunc() outside of the class definition, maybe even in another module?

Output: Yes. You can define a function outside of a class and then use it in the class body as a method: def func(self): print("func") class MyClass: myMethod = func You can also add a function to a class after it has been defined: class MyClass: pass def func(self): print("func") MyClass.myMethod = func You can define the function and the class in different modules if you want, but I'd advise against defining the class in one module then importing it in another and adding methods to it dynamically (as in my second example), because then you'd have surprisingly different behaviour from the class depending on whether or not another module has been imported. I would point out that while this is possible in Python, it's a bit unusual. You mention in a comment that "users are allowed to add more" methods. That sounds odd. If you're writing a library you probably don't want users of the library to add methods dynamically to classes in the library. It's more normal for users of a library to create their own subclass that inherits from your class than to change yours directly. I'd also add a reminder that functions don't have to be in classes at all. Python isn't like Java or C# and you can just have functions that aren't part of any class. If you want to group together functions you can just put them together in the same module, and you can nest modules inside packages. Only use classes when you need to create a new data type, not just to group functions together.


Instruction:
How to write python lambda with multiple lines?

In python, how can you write a lambda function taking multiple lines. I tried d = lambda x: if x: return 1 else return 2 but I am getting errors...

Output: Use def instead. def d(x): if x: return 1 else: return 2 All python functions are first order objects (they can be assigned to variables and passed around as arguments), lambda is just a convenient way to make short ones. In general, you are better off using a normal function definition if it becomes anything beyond one line of simple code. Even then, in fact, if you are assigning it to a name, I would always use def over lambda (something PEP 8 explicitly recommends as it improves debugging). lambda is really only a good idea when defining short functions that can be placed inline into the function call, for example key functions for use with sorted(). Note that, in your case, a ternary operator would do the job (lambda x: 1 if x else 2), but I'm presuming this is a simplified case and you are talking about cases where it's not reasonable to use a single expression. (As a code golf note, this could also be done in less code as lambda x: 2-bool(x) - of course, that's highly unreadable and a bad idea.)


Instruction:
How to install python with conda?

I'm trying to install python 3.9 in a conda enviroment. I tried creating a new conda env using the following command, conda create --name myenv python=3.9 But I got an error saying package not found because python 3.9 is not yet released So, I manually created a folder in envs folder and tried to list all envs. But I couldn't get the manually created new environment. So, how do I install python 3.9 in a conda env with all functionalities like pip working?

Output: To create python 3.11 conda environment use the following command conda create -n py311 python=3.11 py311 - environment name Update 3 To create python 3.10 conda environment use the following command conda create -n py310 python=3.10 py310 - environment name Update 2 You can now directly create python 3.9 environment using the following command conda create -n py39 python=3.9 py39 - environment name Update 1 Python 3.9 is now available in conda-forge. To download the tar file - https://anaconda.org/conda-forge/python/3.9.0/download/linux-64/python-3.9.0-h852b56e_0_cpython.tar.bz2 Anaconda Page - https://anaconda.org/conda-forge/python As pointed out in the comments, python 3.9 is not yet there on any channels. So, it cannot be install yet via conda. Instead, you can download the python 3.9 executable and install it. Once the installation is done, a new executable will be created for python 3.9 and pip 3.9 will be created. Python: python3.7 python3.7-config python3.7m python3.7m-config python3.9 python3.9-config pip pip pip3 pip3.7 pip3.8 pip3.9 pipreqs In order to install ipython for python 3.9, pip3.9 install ipython


Instruction:
A comparison between fastparquet and pyarrow?

After some searching I failed to find a thorough comparison of fastparquet and pyarrow. I found this blog post (a basic comparison of speeds). and a github discussion that claims that files created with fastparquet do not support AWS-athena (btw is it still the case?) when/why would I use one over the other? what are the major advantages and disadvantages ? my specific use case is processing data with dask writing it to s3 and then reading/analyzing it with AWS-athena.

Output: In 2024 the decision should be obvious: use pyarrow instead of fastparquet: Pandas 3.0 will require pyarrow What’s new in 2.1.0 (Aug 30, 2023). fastparquet is deprecated in Dask 2024.1.0, and "users should migrate to the pyarrow engine" fastparquet is from the dask team. If dask itself does not want to use it, why should you? See the discussion in dask / #8900. Some picks from there: In our recent parquet benchmarking and resilience testing we generally found the pyarrow engine would scale to larger datasets better than the fastparquet engine, and more test cases would complete successfully when run with pyarrow than with fastparquet. The pyarrow library has a larger development team maintaining it and seems to have more community buy-in going forward.


Instruction:
Direct assignment to the forward side of a many-to-many set is prohibited. Use emails_for_help.set() instead

I am new to Django and didn't find any reference regarding this issue. I am getting this error when i use many to many field in Django model (models.py). I guess the issue is assigning m2m field in view(views.py) from form(forms.py). How to assign m2m field in view? (Django version 2.0, python - 3.5) models.py class User(AbstractUser): username=models.CharField(max_length=20) email = models.EmailField(_('email address'), unique=True) class Setupuser(models.Model): organization=models.CharField(max_length=200,blank=False,null=True) emails_for_help = models.ManyToManyField(User) views.py class Set_user(FormView): template_name="pkm_templates/set_up_user.html" form_class = Set_User_Form success_url = '/thanks/' def form_valid(self, form): org = form.cleaned_data.get('organization') emails = form.cleaned_data.get("emails_for_help") instance = Setupuser(organization=org,emails_for_help=emails) instance.save() return redirect("/") forms.py class Set_User_Form(ModelForm): emails_for_help = forms.ModelMultipleChoiceField( queryset=User.objects.all(), widget=forms.CheckboxSelectMultiple ) class Meta: model = Setupuser fields = ["organization","emails_for_help"]

Output: You need to get the User object and then add it to emails_for_help field. You can't add an object to ManyToManyField when creating an instance. Have a look at the doc. class Set_user(FormView): template_name="pkm_templates/set_up_user.html" form_class = Set_User_Form success_url = '/thanks/' def form_valid(self, form): org = form.cleaned_data.get('organization') emails = form.cleaned_data.get("share_email_with") users = User.objects.filter(email__in=emails) instance = Setupuser.objects.create(organization=org) for user in users: instance.emails_for_help.add(user) return redirect("/") Another way of doing this is to use .set(). class Set_user(FormView): template_name="pkm_templates/set_up_user.html" form_class = Set_User_Form success_url = '/thanks/' def form_valid(self, form): org = form.cleaned_data.get('organization') emails = form.cleaned_data.get("share_email_with") users = User.objects.filter(email__in=emails) instance = Setupuser.objects.create(organization=org) instance.emails_for_help.set(users) return redirect("/") Or you can simply use .add() to add arbitrary number of objects. class Set_user(FormView): template_name="pkm_templates/set_up_user.html" form_class = Set_User_Form success_url = '/thanks/' def form_valid(self, form): org = form.cleaned_data.get('organization') emails = form.cleaned_data.get("share_email_with") users = User.objects.filter(email__in=emails) instance = Setupuser.objects.create(organization=org) instance.emails_for_help.add(*users) return redirect("/")


Instruction:
How can I get the IP address from a NIC (network interface controller) in Python?

When an error occurs in a Python script on Unix, an email is sent. I have been asked to add {Testing Environment} to the subject line of the email if the IP address is 192.168.100.37 which is the testing server. This way we can have one version of a script and a way to tell if the email is coming from messed up data on the testing server. However, when I google I keep finding this code: import socket socket.gethostbyname(socket.gethostname()) However, that's giving me the IP address of 127.0.1.1. When I use ifconfig I get this eth0 Link encap:Ethernet HWaddr 00:1c:c4:2c:c8:3e inet addr:192.168.100.37 Bcast:192.168.100.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:75760697 errors:0 dropped:411180 overruns:0 frame:0 TX packets:23166399 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:59525958247 (59.5 GB) TX bytes:10142130096 (10.1 GB) Interrupt:19 Memory:f0500000-f0520000 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:25573544 errors:0 dropped:0 overruns:0 frame:0 TX packets:25573544 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:44531490070 (44.5 GB) TX bytes:44531490070 (44.5 GB) Firstly, I don't know where it got 127.0.1.1 from, but either way that's not what I want. When I google I keep coming to the same syntax, Bash scripts or netifaces and I'm trying to use standard libraries. So how can I get the IP address of eth0 in Python?

Output: Two methods: Method #1 (use external package) You need to ask for the IP address that is bound to your eth0 interface. This is available from the netifaces package import netifaces as ni ip = ni.ifaddresses('eth0')[ni.AF_INET][0]['addr'] print(ip) # should print "192.168.100.37" You can also get a list of all available interfaces via ni.interfaces() Method #2 (no external package) Here's a way to get the IP address without using a python package: import socket import fcntl import struct def get_ip_address(ifname): s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) return socket.inet_ntoa(fcntl.ioctl( s.fileno(), 0x8915, # SIOCGIFADDR struct.pack('256s', ifname[:15]) )[20:24]) get_ip_address('eth0') # '192.168.0.110' Note: detecting the IP address to determine what environment you are using is quite a hack. Almost all frameworks provide a very simple way to set/modify an environment variable to indicate the current environment. Try and take a look at your documentation for this. It should be as simple as doing if app.config['ENV'] == 'production': # send production email else: # send development email


Instruction:
How to read a CSV file from a URL with Python?

when I do curl to a API call link http://example.com/passkey=wedsmdjsjmdd curl 'http://example.com/passkey=wedsmdjsjmdd' I get the employee output data on a csv file format, like: "Steve","421","0","421","2","","","","","","","","","421","0","421","2" how can parse through this using python. I tried: import csv cr = csv.reader(open('http://example.com/passkey=wedsmdjsjmdd',"rb")) for row in cr: print row but it didn't work and I got an error http://example.com/passkey=wedsmdjsjmdd No such file or directory: Thanks!

Output: You need to replace open with urllib.urlopen or urllib2.urlopen. e.g. import csv import urllib2 url = 'http://winterolympicsmedals.com/medals.csv' response = urllib2.urlopen(url) cr = csv.reader(response) for row in cr: print row This would output the following Year,City,Sport,Discipline,NOC,Event,Event gender,Medal 1924,Chamonix,Skating,Figure skating,AUT,individual,M,Silver 1924,Chamonix,Skating,Figure skating,AUT,individual,W,Gold ... The original question is tagged "python-2.x", but for a Python 3 implementation (which requires only minor changes) see below.


Instruction:
Send Outlook Email Via Python?

I am using Outlook 2003. What is the best way to send email (through Outlook 2003) using Python?

Output: For a solution that uses outlook see TheoretiCAL's answer. Otherwise, use the smtplib that comes with python. Note that this will require your email account allows smtp, which is not necessarily enabled by default. SERVER = "smtp.example.com" FROM = "yourEmail@example.com" TO = ["listOfEmails"] # must be a list SUBJECT = "Subject" TEXT = "Your Text" # Prepare actual message message = """From: %s\r\nTo: %s\r\nSubject: %s\r\n\ %s """ % (FROM, ", ".join(TO), SUBJECT, TEXT) # Send the mail import smtplib server = smtplib.SMTP(SERVER) server.sendmail(FROM, TO, message) server.quit() EDIT: this example uses reserved domains like described in RFC2606 SERVER = "smtp.example.com" FROM = "johnDoe@example.com" TO = ["JaneDoe@example.com"] # must be a list SUBJECT = "Hello!" TEXT = "This is a test of emailing through smtp of example.com." # Prepare actual message message = """From: %s\r\nTo: %s\r\nSubject: %s\r\n\ %s """ % (FROM, ", ".join(TO), SUBJECT, TEXT) # Send the mail import smtplib server = smtplib.SMTP(SERVER) server.login("MrDoe", "PASSWORD") server.sendmail(FROM, TO, message) server.quit() For it to actually work with gmail, Mr. Doe will need to go to the options tab in gmail and set it to allow smtp connections. Note the addition of the login line to authenticate to the remote server. The original version does not include this, an oversight on my part.


Instruction:
How can I run the FastAPI server using Pycharm?

I have a simple API function as below, from fastapi import FastAPI app = FastAPI() @app.get("/") async def read_root(): return {"Hello": "World"} I am starting the server using uvicorn command as, uvicorn main:app Since we are not calling any python file directly, it is not possible to call uvicorn command from Pycharm. So, How can I run the fast-api server using Pycharm?

Output: Method-1: Run FastAPI by calling uvicorn.run(...) In this case, your minimal code will be as follows, # main.py import uvicorn from fastapi import FastAPI app = FastAPI() @app.get("/") async def read_root(): return {"Hello": "World"} if __name__ == "__main__": uvicorn.run(app, host="0.0.0.0", port=8000) Normally, you'll start the server by running the following command, python main.py Pycharm Setup For this setup, and now, you can set the script path in Pycharm's config Notes Script Path: path to the FastAPI script Python Interpreter: Choose your interpreter/virtual environment Working Directory: Your FastAPI project root Method-2: Run FastAPI by calling uvicorn command In this case, your minimal code will be as follows, # main.py from fastapi import FastAPI app = FastAPI() @app.get("/") async def read_root(): return {"Hello": "World"} Normally, you'll start the server by running the following command, uvicorn main:app --reload Pycharm Setup For this setup, and now, you can set the script path in Pycharm's config Notes Module name: set to uvicorn [Optional] Script: Path to uvicorn binary. You will get the path by executing the command, which uvicorn , inside your environment. (See this image) Parameters: The actual parameters of uvicorn command Python Interpreter: Choose your interpreter/virtual environment Working Directory: Your FastAPI project root


Instruction:
Error while downloading the requirements using pip install (setup command: use_2to3 is invalid.)

version pip 21.2.4 python 3.6 The command: pip install -r requirements.txt The content of my requirements.txt: mongoengine==0.19.1 numpy==1.16.2 pylint pandas==1.1.5 fawkes The command is failing with this error ERROR: Command errored out with exit status 1: command: /Users/*/Desktop/ml/*/venv/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/kn/0y92g7x55qs7c42tln4gwhtm0000gp/T/pip-install-soh30mel/mongoengine_89e68f8427244f1bb3215b22f77a619c/setup.py'"'"'; __file__='"'"'/private/var/folders/kn/0y92g7x55qs7c42tln4gwhtm0000gp/T/pip-install-soh30mel/mongoengine_89e68f8427244f1bb3215b22f77a619c/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/kn/0y92g7x55qs7c42tln4gwhtm0000gp/T/pip-pip-egg-info-97994d6e cwd: /private/var/folders/kn/0y92g7x55qs7c42tln4gwhtm0000gp/T/pip-install-soh30mel/mongoengine_89e68f8427244f1bb3215b22f77a619c/ Complete output (1 lines): error in mongoengine setup command: use_2to3 is invalid. ---------------------------------------- WARNING: Discarding https://*/pypi/packages/mongoengine-0.19.1.tar.gz#md5=68e613009f6466239158821a102ac084 (from https://*/pypi/simple/mongoengine/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output. ERROR: Could not find a version that satisfies the requirement mongoengine==0.19.1 (from versions: 0.15.0, 0.19.1) ERROR: No matching distribution found for mongoengine==0.19.1

Output: It looks like setuptools>=58 breaks support for use_2to3: setuptools changelog for v58 So you should update setuptools to setuptools<58 or avoid using packages with use_2to3 in the setup parameters. I was having the same problem, pip==19.3.1


Instruction:
Giving graphs a subtitle

I want to give my graph a title in big 18pt font, then a subtitle below it in smaller 10pt font. How can I do this in matplotlib? It appears the title() function only takes one single string with a single fontsize attribute. There has to be a way to do this, but how?

Output: I don't think there is anything built-in, but you can do it by leaving more space above your axes and using figtext: axes([.1,.1,.8,.7]) figtext(.5,.9,'Foo Bar', fontsize=18, ha='center') figtext(.5,.85,'Lorem ipsum dolor sit amet, consectetur adipiscing elit',fontsize=10,ha='center') ha is short for horizontalalignment.


Instruction:
Pythonic way to check if something exists?

I was wondering if there is a pythonic way to check if something does not exist. Here's how I do it if its true: var = 1 if var: print 'it exists' but when I check if something does not exist, I often do something like this: var = 2 if var: print 'it exists' else: print 'nope it does not' Seems like a waste if all I care about is kn Is there a way to check if something does not exist without the else?

Output: EAFP style, "easier to ask forgiveness than permission": try: var except NameError: var_exists = False else: var_exists = True LBYL style, "look before you leap": var_exists = 'var' in locals() or 'var' in globals() Prefer the first style (EAFP) when coding in Python, because it is generally more reliable.


Instruction:
Pandas: Subtracting two date columns and the result being an integer

I have two columns in a Pandas data frame that are dates. I am looking to subtract one column from another and the result being the difference in numbers of days as an integer. A peek at the data: df_test.head(10) Out[20]: First_Date Second Date 0 2016-02-09 2015-11-19 1 2016-01-06 2015-11-30 2 NaT 2015-12-04 3 2016-01-06 2015-12-08 4 NaT 2015-12-09 5 2016-01-07 2015-12-11 6 NaT 2015-12-12 7 NaT 2015-12-14 8 2016-01-06 2015-12-14 9 NaT 2015-12-15 I have created a new column successfully with the difference: df_test['Difference'] = df_test['First_Date'].sub(df_test['Second Date'], axis=0) df_test.head() Out[22]: First_Date Second Date Difference 0 2016-02-09 2015-11-19 82 days 1 2016-01-06 2015-11-30 37 days 2 NaT 2015-12-04 NaT 3 2016-01-06 2015-12-08 29 days 4 NaT 2015-12-09 NaT However I am unable to get a numeric version of the result: df_test['Difference'] = df_test[['Difference']].apply(pd.to_numeric) df_test.head() Out[25]: First_Date Second Date Difference 0 2016-02-09 2015-11-19 7.084800e+15 1 2016-01-06 2015-11-30 3.196800e+15 2 NaT 2015-12-04 NaN 3 2016-01-06 2015-12-08 2.505600e+15 4 NaT 2015-12-09 NaN

Output: How about: df_test['Difference'] = (df_test['First_Date'] - df_test['Second Date']).dt.days This will return difference as int if there are no missing values(NaT) and float if there is. Pandas have a rich documentation on Time series / date functionality and Time deltas


Instruction:
How do I plot Shapely polygons and objects using Matplotlib?

I want to use Shapely for my computational geometry project. I need to be able to visualize and display polygons, lines, and other geometric objects for this. I've tried to use Matplotlib for this but I am having trouble with it. from shapely.geometry import Polygon import matplotlib.pyplot as plt polygon1 = Polygon([(0,5), (1,1), (3,0), ]) plt.plot(polygon1) plt.show() I would like to be able to display this polygon in a plot. How would I change my code to do this?

Output: Use: import matplotlib.pyplot as plt x,y = polygon1.exterior.xy plt.plot(x,y) Or, more succinctly: plt.plot(*polygon1.exterior.xy)


Instruction:
How to simply add a column level to a pandas dataframe

let's say I have a dataframe that looks like this: df = pd.DataFrame({'A': range(5), 'B': range(5)}, index=list('abcde')) df Out[92]: A B a 0 0 b 1 1 c 2 2 d 3 3 e 4 4 Asumming that this dataframe already exist, how can I simply add a level 'C' to the column index so I get this: df Out[92]: A B C C a 0 0 b 1 1 c 2 2 d 3 3 e 4 4 I saw SO anwser like this python/pandas: how to combine two dataframes into one with hierarchical column index? but this concat different dataframe instead of adding a column level to an already existing dataframe.

Output: As suggested by @StevenG himself, a better answer: df.columns = pd.MultiIndex.from_product(df.columns.levels + [['C']]) print(df) # A B # C C # a 0 0 # b 1 1 # c 2 2 # d 3 3 # e 4 4


Instruction:
Call to operating system to open url?

What can I use to call the OS to open a URL in whatever browser the user has as default? Not worried about cross-OS compatibility; if it works in linux thats enough for me!

Output: Here is how to open the user's default browser with a given url: import webbrowser url = "https://www.google.com/" webbrowser.open(url, new=0, autoraise=True) Here is the documentation about this functionality. It's part of Python's stdlibs: http://docs.python.org/library/webbrowser.html I have tested this successfully on Linux, Ubuntu 10.10.


Instruction:
Pandas: sum up multiple columns into one column without last column

If I have a dataframe similar to this one Apples Bananas Grapes Kiwis 2 3 nan 1 1 3 7 nan nan nan 2 3 I would like to add a column like this Apples Bananas Grapes Kiwis Fruit Total 2 3 nan 1 6 1 3 7 nan 11 nan nan 2 3 5 I guess you could use df['Apples'] + df['Bananas'] and so on, but my actual dataframe is much larger than this. I was hoping a formula like df['Fruit Total']=df[-4:-1].sum could do the trick in one line of code. That didn't work however. Is there any way to do it without explicitly summing up all columns?

Output: You can first select by iloc and then sum: df['Fruit Total']= df.iloc[:, -4:-1].sum(axis=1) print (df) Apples Bananas Grapes Kiwis Fruit Total 0 2.0 3.0 NaN 1.0 5.0 1 1.0 3.0 7.0 NaN 11.0 2 NaN NaN 2.0 3.0 2.0 For sum all columns use: df['Fruit Total']= df.sum(axis=1)


Instruction:
What is the max length of a Python string?

If it is environment-independent, what is the theoretical maximum number of characters in a Python string?

Output: With a 64-bit Python installation, and (say) 64 GB of memory, a Python string of around 63 GB should be quite feasible, if not maximally fast. If you can upgrade your memory beyond 64 GB, your maximum feasible strings should get proportionally longer. (I don't recommend relying on virtual memory to extend that by much, or your runtimes will get simply ridiculous;-). With a typical 32-bit Python installation, the total memory you can use in your application is limited to something like 2 or 3 GB (depending on OS and configuration), so the longest strings you can use will be much smaller than in 64-bit installations with high amounts of RAM.


Instruction:
How to check if a datetime object is localized with pytz?

I want to store a datetime object with a localized UTC timezone. The method that stores the datetime object can be given a non-localized datetime (naive) object or an object that already has been localized. How do I determine if localization is needed? Code with missing if condition: class MyClass: def set_date(self, d): # what do i check here? # if(d.tzinfo): self.date = d.astimezone(pytz.utc) # else: self.date = pytz.utc.localize(d)

Output: How do I determine if localization is needed? From datetime docs: a datetime object d is aware iff: d.tzinfo is not None and d.tzinfo.utcoffset(d) is not None d is naive iff: d.tzinfo is None or d.tzinfo.utcoffset(d) is None Though if d is a datetime object representing time in UTC timezone then you could use in both cases: self.date = d.replace(tzinfo=pytz.utc) It works regardless d is timezone-aware or naive. Note: don't use datetime.replace() method with a timezone with a non-fixed utc offset (it is ok to use it with UTC timezone but otherwise you should use tz.localize() method).


Instruction:
How can I wrap a synchronous function in an async coroutine?

I'm using aiohttp to build an API server that sends TCP requests off to a seperate server. The module that sends the TCP requests is synchronous and a black box for my purposes. So my problem is that these requests are blocking the entire API. I need a way to wrap the module requests in an asynchronous coroutine that won't block the rest of the API. So, just using sleep as a simple example, is there any way to somehow wrap time-consuming synchronous code in a non-blocking coroutine, something like this: async def sleep_async(delay): # After calling sleep, loop should be released until sleep is done yield sleep(delay) return 'I slept asynchronously'

Output: Eventually I found an answer in this thread. The method I was looking for is run_in_executor. This allows a synchronous function to be run asynchronously without blocking an event loop. In the sleep example I posted above, it might look like this: import asyncio from time import sleep async def sleep_async(loop, delay): # None uses the default executor (ThreadPoolExecutor) await loop.run_in_executor(None, sleep, delay) return 'I slept asynchronously' Also see the following answer -> How do we call a normal function where a coroutine is expected?


Instruction:
Sending SOAP request using Python Requests

Is it possible to use Python's requests library to send a SOAP request?

Output: It is indeed possible. Here is an example calling the Weather SOAP Service using plain requests lib: import requests url="http://wsf.cdyne.com/WeatherWS/Weather.asmx?WSDL" #headers = {'content-type': 'application/soap+xml'} headers = {'content-type': 'text/xml'} body = """<?xml version="1.0" encoding="UTF-8"?> <SOAP-ENV:Envelope xmlns:ns0="http://ws.cdyne.com/WeatherWS/" xmlns:ns1="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"> <SOAP-ENV:Header/> <ns1:Body><ns0:GetWeatherInformation/></ns1:Body> </SOAP-ENV:Envelope>""" response = requests.post(url,data=body,headers=headers) print response.content Some notes: The headers are important. Most SOAP requests will not work without the correct headers. application/soap+xml is probably the more correct header to use (but the weatherservice prefers text/xml This will return the response as a string of xml - you would then need to parse that xml. For simplicity I have included the request as plain text. But best practise would be to store this as a template, then you can load it using jinja2 (for example) - and also pass in variables. For example: from jinja2 import Environment, PackageLoader env = Environment(loader=PackageLoader('myapp', 'templates')) template = env.get_template('soaprequests/WeatherSericeRequest.xml') body = template.render() Some people have mentioned the suds library. Suds is probably the more correct way to be interacting with SOAP, but I often find that it panics a little when you have WDSLs that are badly formed (which, TBH, is more likely than not when you're dealing with an institution that still uses SOAP ;) ). You can do the above with suds like so: from suds.client import Client url="http://wsf.cdyne.com/WeatherWS/Weather.asmx?WSDL" client = Client(url) print client ## shows the details of this service result = client.service.GetWeatherInformation() print result Note: when using suds, you will almost always end up needing to use the doctor. Finally, a little bonus for debugging SOAP; TCPdump is your friend. On Mac, you can run TCPdump like so: sudo tcpdump -As 0 This can be helpful for inspecting the requests that actually go over the wire. The above two code snippets are also available as gists: SOAP Request with requests SOAP Request with suds


Instruction:
What exactly is __weakref__ in Python?

Surprisingly, there's no explicit documentation for __weakref__. Weak references are explained here. __weakref__ is also shortly mentioned in the documentation of __slots__. But I could not find anything about __weakref__ itself. What exactly is __weakref__? - Is it just a member acting as a flag: If present, the object may be weakly-referenced? - Or is it a function/variable that can be overridden/assigned to get a desired behavior? How?

Output: __weakref__ is just an opaque object that references all the weak references to the current object. In actual fact it's an instance of weakref (or sometimes weakproxy) which is both a weak reference to the object and part of a doubly linked list to all weak references for that object. It's just an implementation detail that allows the garbage collector to inform weak references that its referent has been collected, and to not allow access to its underlying pointer anymore. The weak reference can't rely on checking the reference count of the object it refers to. This is because that memory may have been reclaimed and is now being used by another object. Best case scenario the VM will crash, worst case the weak reference will allow access to an object it wasn't originally referring to. This is why the garbage collector must inform the weak reference its referent is no longer valid. See weakrefobject.h for the structure and C-API for this object. And the implementation detail is here


Instruction:
Nested dictionary to multiindex dataframe where dictionary keys are column labels

Say I have a dictionary that looks like this: dictionary = {'A' : {'a': [1,2,3,4,5], 'b': [6,7,8,9,1]}, 'B' : {'a': [2,3,4,5,6], 'b': [7,8,9,1,2]}} and I want a dataframe that looks something like this: A B a b a b 0 1 6 2 7 1 2 7 3 8 2 3 8 4 9 3 4 9 5 1 4 5 1 6 2 Is there a convenient way to do this? If I try: In [99]: DataFrame(dictionary) Out[99]: A B a [1, 2, 3, 4, 5] [2, 3, 4, 5, 6] b [6, 7, 8, 9, 1] [7, 8, 9, 1, 2] I get a dataframe where each element is a list. What I need is a multiindex where each level corresponds to the keys in the nested dict and the rows corresponding to each element in the list as shown above. I think I can work a very crude solution but I'm hoping there might be something a bit simpler.

Output: Pandas wants the MultiIndex values as tuples, not nested dicts. The simplest thing is to convert your dictionary to the right format before trying to pass it to DataFrame: >>> reform = {(outerKey, innerKey): values for outerKey, innerDict in dictionary.items() for innerKey, values in innerDict.items()} >>> reform {('A', 'a'): [1, 2, 3, 4, 5], ('A', 'b'): [6, 7, 8, 9, 1], ('B', 'a'): [2, 3, 4, 5, 6], ('B', 'b'): [7, 8, 9, 1, 2]} >>> pandas.DataFrame(reform) A B a b a b 0 1 6 2 7 1 2 7 3 8 2 3 8 4 9 3 4 9 5 1 4 5 1 6 2 [5 rows x 4 columns]


Instruction:
Using a pip cache directory in docker builds

I'm hoping to get my pip install instructions inside my docker builds as fast as possible. I've read many posts explaining how adding your requirements.txt before the rest of the app helps you take advantage of Docker's own image cache if your requirements.txt hasn't changed. But this is no help at all when dependencies do change, even slightly. The next step would be if we could use a consistent pip cache directory. By default, pip will cache downloaded packages in ~/.cache/pip (on Linux), and so if you're ever installing the same version of a module that has been installed before anywhere on the system, it shouldn't need to go and download it again, but instead simply use the cached version. If we could leverage a shared cache directory for docker builds, this could help speed up dependency installs a lot. However, there doesn't appear to be any simple way to mount a volume while running docker build. The build environment seems to be basically impenetrable. I found one article suggesting a genius but complex method of running an rsync server on the host and then, with a hack inside the build to get the host IP, rsyncing the pip cache in from the host. But I'm not relishing the idea of running an rsync server in Jenkins (which isn't the most secure platform at the best of times). Does anyone know if there's any other way to achieve a shared cache volume more simply?

Output: I suggest you to use buildkit, also see this. Dockerfile: # syntax=docker/dockerfile:1 FROM python:3.6-alpine RUN --mount=type=cache,target=/root/.cache/pip pip install pyyaml NOTE: # syntax = docker/dockerfile:experimental is a must，you have to add it at the beginning of Dockerfile to enable this feature. 1. The first execute build: export DOCKER_BUILDKIT=1 docker build --progress=plain -t abc:1 . --no-cache The first log: #9 [stage-0 2/2] RUN --mount=type=cache,target=/root/.cache/pip pip install... #9 digest: sha256:55b70da1cbbe4d424f8c50c0678a01e855510bbda9d26f1ac5b983808f3bf4a5 #9 name: "[stage-0 2/2] RUN --mount=type=cache,target=/root/.cache/pip pip install pyyaml" #9 started: 2019-09-20 03:11:35.296107357 +0000 UTC #9 1.955 Collecting pyyaml #9 3.050 Downloading https://files.pythonhosted.org/packages/e3/e8/b3212641ee2718d556df0f23f78de8303f068fe29cdaa7a91018849582fe/PyYAML-5.1.2.tar.gz (265kB) #9 5.006 Building wheels for collected packages: pyyaml #9 5.007 Building wheel for pyyaml (setup.py): started #9 5.249 Building wheel for pyyaml (setup.py): finished with status 'done' #9 5.250 Created wheel for pyyaml: filename=PyYAML-5.1.2-cp36-cp36m-linux_x86_64.whl size=44104 sha256=867daf35eab43c2d047ad737ea1e9eaeb4168b87501cd4d62c533f671208acaa #9 5.250 Stored in directory: /root/.cache/pip/wheels/d9/45/dd/65f0b38450c47cf7e5312883deb97d065e030c5cca0a365030 #9 5.267 Successfully built pyyaml #9 5.274 Installing collected packages: pyyaml #9 5.309 Successfully installed pyyaml-5.1.2 #9completed: 2019-09-20 03:11:42.221146294 +0000 UTC #9 duration: 6.925038937s From above, you can see the first time, the build will download pyyaml from internet. 2. The second execute build: docker build --progress=plain -t abc:1 . --no-cache The second log: #9 [stage-0 2/2] RUN --mount=type=cache,target=/root/.cache/pip pip install... #9 digest: sha256:55b70da1cbbe4d424f8c50c0678a01e855510bbda9d26f1ac5b983808f3bf4a5 #9 name: "[stage-0 2/2] RUN --mount=type=cache,target=/root/.cache/pip pip install pyyaml" #9 started: 2019-09-20 03:16:58.588157354 +0000 UTC #9 1.786 Collecting pyyaml #9 2.234 Installing collected packages: pyyaml #9 2.270 Successfully installed pyyaml-5.1.2 #9completed: 2019-09-20 03:17:01.933398002 +0000 UTC #9 duration: 3.345240648s From above, you can see the build no longer download package from internet, just use the cache. NOTE, this is not the traditional docker build cache as I have use --no-cache, it's /root/.cache/pip which I mount into build. 3. The third execute build which delete buildkit cache: docker builder prune docker build --progress=plain -t abc:1 . --no-cache The third log: #9 [stage-0 2/2] RUN --mount=type=cache,target=/root/.cache/pip pip install... #9 digest: sha256:55b70da1cbbe4d424f8c50c0678a01e855510bbda9d26f1ac5b983808f3bf4a5 #9 name: "[stage-0 2/2] RUN --mount=type=cache,target=/root/.cache/pip pip install pyyaml" #9 started: 2019-09-20 03:19:07.434792944 +0000 UTC #9 1.894 Collecting pyyaml #9 2.740 Downloading https://files.pythonhosted.org/packages/e3/e8/b3212641ee2718d556df0f23f78de8303f068fe29cdaa7a91018849582fe/PyYAML-5.1.2.tar.gz (265kB) #9 3.319 Building wheels for collected packages: pyyaml #9 3.319 Building wheel for pyyaml (setup.py): started #9 3.560 Building wheel for pyyaml (setup.py): finished with status 'done' #9 3.560 Created wheel for pyyaml: filename=PyYAML-5.1.2-cp36-cp36m-linux_x86_64.whl size=44104 sha256=cea5bc4689e231df7915c2fc3abca225d4ee2e869a7540682aacb6d42eb17053 #9 3.560 Stored in directory: /root/.cache/pip/wheels/d9/45/dd/65f0b38450c47cf7e5312883deb97d065e030c5cca0a365030 #9 3.580 Successfully built pyyaml #9 3.585 Installing collected packages: pyyaml #9 3.622 Successfully installed pyyaml-5.1.2 #9completed: 2019-09-20 03:19:12.530742712 +0000 UTC #9 duration: 5.095949768s From above, you can see if delete buildkit cache, the package download again. In a word, it will give you a shared cache between several times build, and this cache will only be mounted when image build. But, the image self will not have these cache, so avoid a lots of intermediate layer in image. EDIT for folks who are using docker compose and are lazy to read the comments...: You can also do this with docker-compose if you set COMPOSE_DOCKER_CLI_BUILD=1. For example: COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker-compose build – UPDATE according to folk's question 2020/09/02: I don't know from which version (my version now is 19.03.11), if not specify mode for cache directory, the cache won't be reused by next time build. Don't know the detail reason, but you could add mode=0755, to Dockerfile to make it work again: Dockerfile: # syntax = docker/dockerfile:experimental FROM python:3.6-alpine RUN --mount=type=cache,mode=0755,target=/root/.cache/pip pip install pyyaml UPDATE according to folk's question 2023/04/23: Q: Where is the cache exactly on the host? A: The cache on host is maintained by docker with an overlay. You could use next command docker buildx du --verbose and find a entry type Type: exec.cachemount, after that you got the ID: ntpjzcz8hhx31b80nwxji05hn: ID: ntpjzcz8hhx31b80nwxji05hn Created at: 2023-04-23 01:36:41.102680066 +0000 UTC Mutable: true Reclaimable: true Shared: false Size: 3.601MB Description: cached mount /root/.cache/pip from exec /bin/sh -c pip install pyyaml Usage count: 2 Last used: 7 minutes ago Type: exec.cachemount Afterwards, go to /var/lib/docker/overlay2/ntpjzcz8hhx31b80nwxji05hn/diff/cache/wheels to find the cached pyyaml (depends on the ID you got from above). For my station, it looks like next: root@shdebian1:/var/lib/docker/overlay2/ntpjzcz8hhx31b80nwxji05hn/diff/cache/wheels/81/5a/02/b3447894318b70e3cbff3cb4f1a50d9d50a848185358de1d71# ls PyYAML-6.0-cp36-cp36m-linux_x86_64.whl


Instruction:
Python setuptools: How can I list a private repository under install_requires?

I am creating a setup.py file for a project which depends on private GitHub repositories. The relevant parts of the file look like this: from setuptools import setup setup(name='my_project', ..., install_requires=[ 'public_package', 'other_public_package', 'private_repo_1', 'private_repo_2', ], dependency_links=[ 'https://github.com/my_account/private_repo_1/master/tarball/', 'https://github.com/my_account/private_repo_2/master/tarball/', ], ..., ) I am using setuptools instead of distutils because the latter does not support the install_requires and dependency_links arguments per this answer. The above setup file fails to access the private repos with a 404 error - which is to be expected since GitHub returns a 404 to unauthorized requests for a private repository. However, I can't figure out how to make setuptools authenticate. Here are some things I've tried: Use git+ssh:// instead of https:// in dependency_links as I would if installing the repo with pip. This fails because setuptools doesn't recognize this protocol ("unknown url type: git+ssh"), though the distribute documentation says it should. Ditto git+https and git+http. https://<username>:<password>@github.com/... - still get a 404. (This method doesn't work with curl or wget from the command line either - though curl -u <username> <repo_url> -O <output_file_name> does work.) Upgrading setuptools (0.9.7) and virtualenv (1.10) to the latest versions. Also tried installing distribute though this overview says it was merged back into setuptools. Either way, no dice. Currently I just have setup.py print out a warning that the private repos must be downloaded separately. This is obviously less than ideal. I feel like there's something obvious that I'm missing, but can't think what it might be. :) Duplicate-ish question with no answers here.

Output: I was trying to get this to work for installing with pip, but the above was not working for me. From [1] I understood the PEP508 standard should be used, from [2] I retrieved an example which actually does work (at least for my case). Please note; this is with pip 20.0.2 on Python 3.7.4 setup( name='<package>', ... install_requires=[ '<normal_dependency>', # Private repository '<dependency_name> @ git+ssh://git@github.com/<user>/<repo_name>@<branch>', # Public repository '<dependency_name> @ git+https://github.com/<user>/<repo_name>@<branch>', ], ) After specifying my package this way installation works fine (also with -e settings and without the need to specify --process-dependency-links). References [1] https://github.com/pypa/pip/issues/4187 [2] https://github.com/pypa/pip/issues/5566


Instruction:
matching any character including newlines in a Python regex subexpression, not globally

I want to use re.MULTILINE but NOT re.DOTALL, so that I can have a regex that includes both an "any character" wildcard and the normal . wildcard that doesn't match newlines. Is there a way to do this? What should I use to match any character in those instances that I want to include newlines?

Output: To match a newline, or "any symbol" without re.S/re.DOTALL, you may use any of the following: (?s). - the inline modifier group with s flag on sets a scope where all . patterns match any char including line break chars Any of the following work-arounds: [\s\S] [\w\W] [\d\D] The main idea is that the opposite shorthand classes inside a character class match any symbol there is in the input string. Comparing it to (.|\s) and other variations with alternation, the character class solution is much more efficient as it involves much less backtracking (when used with a * or + quantifier). Compare the small example: it takes (?:.|\n)+ 45 steps to complete, and it takes [\s\S]+ just 2 steps. See a Python demo where I am matching a line starting with 123 and up to the first occurrence of 3 at the start of a line and including the rest of that line: import re text = """abc 123 def 356 more text...""" print( re.findall(r"^123(?s:.*?)^3.*", text, re.M) ) # => ['123\ndef\n356'] print( re.findall(r"^123[\w\W]*?^3.*", text, re.M) ) # => ['123\ndef\n356']


Instruction:
How to fix pylint logging-not-lazy?

I am using prospector to examine my code. Pylint returned a logging-not-lazy warning about my debug message. Line: 31 pylint: logging-not-lazy / Specify string format arguments as logging function parameters (col 16) Line: 42 pylint: logging-not-lazy / Specify string format arguments as logging function parameters (col 12) My code is: logging.debug("detect mimetypes faild because %s" % e ) How do I fix logging-not-lazy in pylint?

Output: This means, that you should rewrite your code as: logging.debug("detect mimetypes faild because %s", e) According to https://docs.python.org/2/library/logging.html Logger.debug(msg, *args, **kwargs) ... Logs a message with level DEBUG on this logger. The msg is the message format string, and the args are the arguments which are merged into msg using the string formatting operator. (Note that this means that you can use keywords in the format string, together with a single dictionary argument.) ...


Instruction:
Associativity of "in" in Python?

I'm making a Python parser, and this is really confusing me: >>> 1 in [] in 'a' False >>> (1 in []) in 'a' TypeError: 'in <string>' requires string as left operand, not bool >>> 1 in ([] in 'a') TypeError: 'in <string>' requires string as left operand, not list How exactly does in work in Python, with regards to associativity, etc.? Why do no two of these expressions behave the same way?

Output: 1 in [] in 'a' is evaluated as (1 in []) and ([] in 'a').¹ Since the first condition (1 in []) is False, the whole condition is evaluated as False; ([] in 'a') is never actually evaluated, so no error is raised. We can see how Python executes each statement using the dis module: >>> from dis import dis >>> dis("1 in [] in 'a'") 1 0 LOAD_CONST 0 (1) 2 BUILD_LIST 0 4 DUP_TOP 6 ROT_THREE 8 CONTAINS_OP 0 # `in` is the contains operator 10 JUMP_IF_FALSE_OR_POP 18 # skip to 18 if the first # comparison is false 12 LOAD_CONST 1 ('a') # 12-16 are never executed 14 CONTAINS_OP 0 # so no error here (14) 16 RETURN_VALUE >> 18 ROT_TWO 20 POP_TOP 22 RETURN_VALUE >>> dis("(1 in []) in 'a'") 1 0 LOAD_CONST 0 (1) 2 LOAD_CONST 1 (()) 4 CONTAINS_OP 0 # perform 1 in [] 6 LOAD_CONST 2 ('a') # now load 'a' 8 CONTAINS_OP 0 # check if result of (1 in []) is in 'a' # throws Error because (False in 'a') # is a TypeError 10 RETURN_VALUE >>> dis("1 in ([] in 'a')") 1 0 LOAD_CONST 0 (1) 2 BUILD_LIST 0 4 LOAD_CONST 1 ('a') 6 CONTAINS_OP 0 # perform ([] in 'a'), which is # incorrect, so it throws a TypeError 8 CONTAINS_OP 0 # if no Error then this would # check if 1 is in the result of ([] in 'a') 10 RETURN_VALUE Except that [] is only evaluated once. This doesn't matter in this example but if you (for example) replaced [] with a function that returned a list, that function would only be called once (at most). The documentation explains also this.


Instruction:
Difference between dict and set (python)

So, I know that this, a = {} # dict constructs an empty dictionary. Now, I also picked up that this, b = {1, 2, 3} # set creates a set. This can easily be verified, as, >>>print(type(a)) <class 'dict'> >>>print(type(b)) <class 'set'> While I understand what it does, I fail to see why we use 'set notation' for empty dictionaries. I tried to find some more information about the logic behind this in the set and dict sections of the manual, but sadly, I got nothing out of it. Could anyone explain to me why we do this in this way? Is it for historical reasons, or am I missing something blatantly obvious?

Output: There were no set literals in Python 2, historically curly braces were only used for dictionaries. Sets could be produced from lists (or any iterables): set([1, 2, 3]) set([i for i in range(1, 3)]) To create an empty set, use my_set = set() Python 3 introduced set literals and comprehensions (see PEP-3100) which allowed us to avoid intermediate lists: {1, 2, 3} {i for i in range(1, 3)} The empty set form, however, was reserved for dictionaries due to backwards compatibility. References from [Python-3000] sets in P3K? states: I'm sure we can work something out --- I agree, {} for empty set and {:} for empty dict would be ideal, were it not for backward compatibility. I liked the "special empty object" idea when I first wrote the PEP (i.e., have {} be something that could turn into either a set or dict), but one of the instructors here convinced me that it would just lead to confusion in newcomers' minds (as well as being a pain to implement). The following message describes these rules better: I think Guido had the best solution. Use set() for empty sets, use {} for empty dicts, use {genexp} for set comprehensions/displays, use {1,2,3} for explicit set literals, and use {k1:v1, k2:v2} for dict literals. We can always add {/} later if demand exceeds distaste.


Instruction:
Importing a library from (or near) a script with the same name raises "AttributeError: module has no attribute" or an ImportError or NameError

I have a script named requests.py that needs to use the third-party requests package. The script either can't import the package, or can't access its functionality. Why isn't this working, and how do I fix it? Trying a plain import and then using the functionality results in an AttributeError: import requests res = requests.get('http://www.google.ca') print(res) Traceback (most recent call last): File "/Users/me/dev/rough/requests.py", line 1, in <module> import requests File "/Users/me/dev/rough/requests.py", line 3, in <module> requests.get('http://www.google.ca') AttributeError: module 'requests' has no attribute 'get' In more recent versions of Python, the error message instead reads AttributeError: partially initialized module 'requests' has no attribute 'get' (most likely due to a circular import). Using from-import of a specific name results in an ImportError: from requests import get res = get('http://www.google.ca') print(res) Traceback (most recent call last): File "requests.py", line 1, in <module> from requests import get File "/Users/me/dev/rough/requests.py", line 1, in <module> from requests import get ImportError: cannot import name 'get' In more recent versions of Python, the error message instead reads ImportError: cannot import name 'get' from partially initialized module 'requests' (most likely due to a circular import) (/Users/me/dev/rough/requests.py). Using from-import for a module inside the package results in a different ImportError: from requests.auth import AuthBase Traceback (most recent call last): File "requests.py", line 1, in <module> from requests.auth import AuthBase File "/Users/me/dev/rough/requests.py", line 1, in <module> from requests.auth import AuthBase ImportError: No module named 'requests.auth'; 'requests' is not a package Using a star-import and then using the functionality raises a NameError: from requests import * res = get('http://www.google.ca') print(res) Traceback (most recent call last): File "requests.py", line 1, in <module> from requests import * File "/Users/me/dev/rough/requests.py", line 3, in <module> res = get('http://www.google.ca') NameError: name 'get' is not defined From Python 3.13 onwards the error message becomes very clear, from the documentation: A common mistake is to write a script with the same name as a standard library module. When this results in errors, we now display a more helpful error message: $ python random.py Traceback (most recent call last): File "/home/me/random.py", line 1, in <module> import random File "/home/me/random.py", line 3, in <module> print(random.randint(5)) ^^^^^^^^^^^^^^ AttributeError: module 'random' has no attribute 'randint' (consider renaming '/home/me/random.py' since it has the same name as the standard library module named 'random' and prevents importing that standard library module) Similarly, if a script has the same name as a third-party module that it attempts to import and this results in errors, we also display a more helpful error message: $ python numpy.py Traceback (most recent call last): File "/home/me/numpy.py", line 1, in <module> import numpy as np File "/home/me/numpy.py", line 3, in <module> np.array([1, 2, 3]) ^^^^^^^^ AttributeError: module 'numpy' has no attribute 'array' (consider renaming '/home/me/numpy.py' if it has the same name as a library you intended to import) For cases where you name your module the same as an existing one on purpose and want to handle that situation, see How can I import from the standard library, when my project has a module with the same name? (How can I control where Python looks for modules?)

Output: This happens because your local module named requests.py shadows the installed requests module you are trying to use. The current directory is prepended to sys.path, so the local name takes precedence over the installed name. An extra debugging tip when this comes up is to look at the Traceback carefully, and realize that the name of your script in question is matching the module you are trying to import: Notice the name you used in your script: File "/Users/me/dev/rough/requests.py", line 1, in <module> The module you are trying to import: requests Rename your module to something else to avoid the name collision. Python may generate a requests.pyc file next to your requests.py file (in the __pycache__ directory in Python 3). Remove that as well after your rename, as the interpreter will still reference that file, re-producing the error. However, the pyc file in __pycache__ should not affect your code if the py file has been removed. In the example, renaming the file to my_requests.py, removing requests.pyc, and running again successfully prints <Response [200]>. Note: This doesn't only happen when naming your file as the module you are trying to import. This can also happen if you name your file the same as a module imported by a module you import directly. For example, having a file called copy.py and trying to import pandas from there, will give ImportError: cannot import name 'copy' from 'copy' That is because pandas imports copy. There is no magic solution here as you can't know all the modules' names in the world, but a rule of thumb is to try to make names of modules as unique as possible and try to change the name whenever you get such error.


Instruction:
TypeError: list indices must be integers or slices, not str

I've got two lists that I want to merge into a single array and finally put it in a csv file. How I can avoid this error : def fill_csv(self, array_urls, array_dates, csv_file_path): result_array = [] array_length = str(len(array_dates)) # We fill the CSV file file = open(csv_file_path, "w") csv_file = csv.writer(file, delimiter=';', lineterminator='\n') # We merge the two arrays in one for i in array_length: result_array[i][0].append(array_urls[i]) result_array[i][1].append(array_dates[i]) i += 1 csv_file.writerows(result_array) And got : File "C:\Users\--\gcscan.py", line 63, in fill_csv result_array[i][0].append(array_urls[i]) TypeError: list indices must be integers or slices, not str How can my count work ?

Output: First, array_length should be an integer and not a string: array_length = len(array_dates) Second, your for loop should be constructed using range: for i in range(array_length): # Use `xrange` for python 2. Third, i will increment automatically, so delete the following line: i += 1 Note, one could also just zip the two lists given that they have the same length: import csv dates = ['2020-01-01', '2020-01-02', '2020-01-03'] urls = ['www.abc.com', 'www.cnn.com', 'www.nbc.com'] csv_file_patch = '/path/to/filename.csv' with open(csv_file_patch, 'w') as fout: csv_file = csv.writer(fout, delimiter=';', lineterminator='\n') result_array = zip(dates, urls) csv_file.writerows(result_array)


Instruction:
How to validate an Email address in Django?

I have written a function for adding emails to newsletter base. Until I've added checking validity of sent email it was working flawlessly. Now each time I'm getting "Wrong email" in return. Can anybody see any errors here ? The regex used is : \b[\w\.-]+@[\w\.-]+\.\w{2,4}\b and it is 100% valid (http://gskinner.com/RegExr/), but I may be using it wrong, or it may be some logic error : def newsletter_add(request): if request.method == "POST": try: e = NewsletterEmails.objects.get(email = request.POST['email']) message = _(u"Email is already added.") type = "error" except NewsletterEmails.DoesNotExist: if validateEmail(request.POST['email']): try: e = NewsletterEmails(email = request.POST['email']) except DoesNotExist: pass message = _(u"Email added.") type = "success" e.save() else: message = _(u"Wrong email") type = "error" import re def validateEmail(email): if len(email) > 6: if re.match('\b[\w\.-]+@[\w\.-]+\.\w{2,4}\b', email) != None: return 1 return 0

Output: UPDATE 2017: the code below is 7 years old and was since modified, fixed and expanded. For anyone wishing to do this now, the correct code lives around here. Here is part of django.core.validators you may find interesting :) class EmailValidator(RegexValidator): def __call__(self, value): try: super(EmailValidator, self).__call__(value) except ValidationError, e: # Trivial case failed. Try for possible IDN domain-part if value and u'@' in value: parts = value.split(u'@') domain_part = parts[-1] try: parts[-1] = parts[-1].encode('idna') except UnicodeError: raise e super(EmailValidator, self).__call__(u'@'.join(parts)) else: raise email_re = re.compile( r"(^[-!#$%&'*+/=?^_`{}|~0-9A-Z]+(\.[-!#$%&'*+/=?^_`{}|~0-9A-Z]+)*" # dot-atom r'|^"([\001-\010\013\014\016-\037!#-\[\]-\177]|\\[\001-011\013\014\016-\177])*"' # quoted-string r')@(?:[A-Z0-9](?:[A-Z0-9-]{0,61}[A-Z0-9])?\.)+[A-Z]{2,6}\.?$', re.IGNORECASE) # domain validate_email = EmailValidator(email_re, _(u'Enter a valid e-mail address.'), 'invalid') so if you don't want to use forms and form fields, you can import email_re and use it in your function, or even better - import validate_email and use it, catching possible ValidationError. def validateEmail( email ): from django.core.validators import validate_email from django.core.exceptions import ValidationError try: validate_email( email ) return True except ValidationError: return False And here is Mail::RFC822::Address regexp used in PERL, if you really need to be that paranoid.