Instruction:
How can I replicate rows of a Pandas DataFrame?

My pandas dataframe looks like this: Person ID ZipCode Gender 0 12345 882 38182 Female 1 32917 271 88172 Male 2 18273 552 90291 Female I want to replicate every row 3 times and reset the index to get: Person ID ZipCode Gender 0 12345 882 38182 Female 1 12345 882 38182 Female 2 12345 882 38182 Female 3 32917 271 88172 Male 4 32917 271 88172 Male 5 32917 271 88172 Male 6 18273 552 90291 Female 7 18273 552 90291 Female 8 18273 552 90291 Female I tried solutions such as: pd.concat([df[:5]]*3, ignore_index=True) which adds the rows to the end of the dataframe, instead of having 3 duplicate lines one after the other And: df.reindex(np.repeat(df.index.values, df['ID']), method='ffill') But none of them worked.

Output: Solutions: Use np.repeat: Version 1: Try using np.repeat: newdf = pd.DataFrame(np.repeat(df.values, 3, axis=0)) newdf.columns = df.columns print(newdf) The above code will output: Person ID ZipCode Gender 0 12345 882 38182 Female 1 12345 882 38182 Female 2 12345 882 38182 Female 3 32917 271 88172 Male 4 32917 271 88172 Male 5 32917 271 88172 Male 6 18273 552 90291 Female 7 18273 552 90291 Female 8 18273 552 90291 Female np.repeat repeats the values of df, 3 times. Then we add the columns with assigning new_df.columns = df.columns. Version 2: You could also assign the column names in the first line, like below: newdf = pd.DataFrame(np.repeat(df.values, 3, axis=0), columns=df.columns) print(newdf) The above code will also output: Person ID ZipCode Gender 0 12345 882 38182 Female 1 12345 882 38182 Female 2 12345 882 38182 Female 3 32917 271 88172 Male 4 32917 271 88172 Male 5 32917 271 88172 Male 6 18273 552 90291 Female 7 18273 552 90291 Female 8 18273 552 90291 Female Version 3: You could shorten it with loc and only repeat the index, like below: newdf = df.loc[np.repeat(df.index, 3)].reset_index(drop=True) print(newdf) The above code will also output: Person ID ZipCode Gender 0 12345 882 38182 Female 1 12345 882 38182 Female 2 12345 882 38182 Female 3 32917 271 88172 Male 4 32917 271 88172 Male 5 32917 271 88172 Male 6 18273 552 90291 Female 7 18273 552 90291 Female 8 18273 552 90291 Female I use reset_index to replace the index with monotonic indexes (0, 1, 2, 3, 4...). Without np.repeat: Version 4: You could use the built-in pd.Index.repeat function, like the below: newdf = df.loc[df.index.repeat(3)].reset_index(drop=True) print(newdf) The above code will also output: Person ID ZipCode Gender 0 12345 882 38182 Female 1 12345 882 38182 Female 2 12345 882 38182 Female 3 32917 271 88172 Male 4 32917 271 88172 Male 5 32917 271 88172 Male 6 18273 552 90291 Female 7 18273 552 90291 Female 8 18273 552 90291 Female Remember to add reset_index to line-up the index. Version 5: Or by using concat with sort_index, like below: newdf = pd.concat([df] * 3).sort_index().reset_index(drop=True) print(newdf) The above code will also output: Person ID ZipCode Gender 0 12345 882 38182 Female 1 12345 882 38182 Female 2 12345 882 38182 Female 3 32917 271 88172 Male 4 32917 271 88172 Male 5 32917 271 88172 Male 6 18273 552 90291 Female 7 18273 552 90291 Female 8 18273 552 90291 Female Version 6: You could also use loc with Python list multiplication and sorted, like below: newdf = df.loc[sorted([*df.index] * 3)].reset_index(drop=True) print(newdf) The above code will also output: Person ID ZipCode Gender 0 12345 882 38182 Female 1 12345 882 38182 Female 2 12345 882 38182 Female 3 32917 271 88172 Male 4 32917 271 88172 Male 5 32917 271 88172 Male 6 18273 552 90291 Female 7 18273 552 90291 Female 8 18273 552 90291 Female Timings: Timing with the following code: import timeit import pandas as pd import numpy as np df = pd.DataFrame({'Person': {0: 12345, 1: 32917, 2: 18273}, 'ID': {0: 882, 1: 271, 2: 552}, 'ZipCode': {0: 38182, 1: 88172, 2: 90291}, 'Gender': {0: 'Female', 1: 'Male', 2: 'Female'}}) def version1(): newdf = pd.DataFrame(np.repeat(df.values, 3, axis=0)) newdf.columns = df.columns def version2(): newdf = pd.DataFrame(np.repeat(df.values, 3, axis=0), columns=df.columns) def version3(): newdf = df.loc[np.repeat(df.index, 3)].reset_index(drop=True) def version4(): newdf = df.loc[df.index.repeat(3)].reset_index(drop=True) def version5(): newdf = pd.concat([df] * 3).sort_index().reset_index(drop=True) def version6(): newdf = df.loc[sorted([*df.index] * 3)].reset_index(drop=True) print('Version 1 Speed:', timeit.timeit('version1()', 'from __main__ import version1', number=20000)) print('Version 2 Speed:', timeit.timeit('version2()', 'from __main__ import version2', number=20000)) print('Version 3 Speed:', timeit.timeit('version3()', 'from __main__ import version3', number=20000)) print('Version 4 Speed:', timeit.timeit('version4()', 'from __main__ import version4', number=20000)) print('Version 5 Speed:', timeit.timeit('version5()', 'from __main__ import version5', number=20000)) print('Version 6 Speed:', timeit.timeit('version6()', 'from __main__ import version6', number=20000)) Output: Version 1 Speed: 9.879425965991686 Version 2 Speed: 7.752138633004506 Version 3 Speed: 7.078321029010112 Version 4 Speed: 8.01169377300539 Version 5 Speed: 19.853051771002356 Version 6 Speed: 9.801617017001263 We can see that Versions 2 & 3 are faster than the others, the reason for this is because they both use the np.repeat function, and numpy functions are very fast because they are implemented with C. Version 3 wins against Version 2 marginally due to the usage of loc instead of DataFrame. Version 5 is significantly slower because of the functions concat and sort_index, since concat copies DataFrames quadratically, which takes longer time. Fastest Version: Version 3.


Instruction:
Extract files from zip without keeping the structure using python ZipFile?

I try to extract all files from .zip containing subfolders in one folder. I want all the files from subfolders extract in only one folder without keeping the original structure. At the moment, I extract all, move the files to a folder, then remove previous subfolders. The files with same names are overwrited. Is it possible to do it before writing files? Here is a structure for example: my_zip/file1.txt my_zip/dir1/file2.txt my_zip/dir1/dir2/file3.txt my_zip/dir3/file4.txt At the end I whish this: my_dir/file1.txt my_dir/file2.txt my_dir/file3.txt my_dir/file4.txt What can I add to this code ? import zipfile my_dir = "D:\\Download\\" my_zip = "D:\\Download\\my_file.zip" zip_file = zipfile.ZipFile(my_zip, 'r') for files in zip_file.namelist(): zip_file.extract(files, my_dir) zip_file.close() if I rename files path from zip_file.namelist(), I have this error: KeyError: "There is no item named 'file2.txt' in the archive"

Output: This opens file handles of members of the zip archive, extracts the filename and copies it to a target file (that's how ZipFile.extract works, without taking care of subdirectories). import os import shutil import zipfile my_dir = r"D:\Download" my_zip = r"D:\Download\my_file.zip" with zipfile.ZipFile(my_zip) as zip_file: for member in zip_file.namelist(): filename = os.path.basename(member) # skip directories if not filename: continue # copy file (taken from zipfile's extract) source = zip_file.open(member) target = open(os.path.join(my_dir, filename), "wb") with source, target: shutil.copyfileobj(source, target)


Instruction:
Python re.sub with a flag does not replace all occurrences

The Python docs say: re.MULTILINE: When specified, the pattern character '^' matches at the beginning of the string and at the beginning of each line (immediately following each newline)... By default, '^' matches only at the beginning of the string... So what's going on when I get the following unexpected result? >>> import re >>> s = """// The quick brown fox. ... // Jumped over the lazy dog.""" >>> re.sub('^//', '', s, re.MULTILINE) ' The quick brown fox.\n// Jumped over the lazy dog.'

Output: Look at the definition of re.sub: re.sub(pattern, repl, string[, count, flags]) The 4th argument is the count, you are using re.MULTILINE (which is 8) as the count, not as a flag. Either use a named argument: re.sub('^//', '', s, flags=re.MULTILINE) Or compile the regex first: re.sub(re.compile('^//', re.MULTILINE), '', s)


Instruction:
Execute Python script within Jupyter notebook using a specific virtualenv

I would like to execute a long running Python script from within a Jupyter notebook so that I can hack on the data structures generated mid-run. The script has many dependencies and command line arguments and is executed with a specific virtualenv. Is it possible to interactively run a Python script inside a notebook from a specified virtualenv (different to that of the Jupyter installation)?

Output: Here's what worked for me (non conda python): (MacOS, brew version of python. if you are working with system python, you may (will) need prepend each command with sudo) First activate virtualenv. If starting afresh then, e.g., you could use virtualenvwrapper: $ pip install virtualenvwrapper $ mkvirtualenv -p python2 py2env $ workon py2env # This will activate virtualenv (py2env)$ # Then install jupyter within the active virtualenv (py2env)$ pip install jupyter # jupyter comes with ipykernel, but somehow you manage to get an error due to ipykernel, then for reference ipykernel package can be installed using: (py2env)$ pip install ipykernel Next, set up the kernel (py2env)$ python -m ipykernel install --user --name py2env --display-name "Python2 (py2env)" then start jupyter notebook (the venv need not be activated for this step) (py2env)$ jupyter notebook # or #$ jupyter notebook In the jupyter notebook dropdown menu: Kernel >> Change Kernel >> <list of kernels> you should see Python2 (py2env) kernel. This also makes it easy to identify python version of kernel, and maintain either side by side. Here is the link to detailed docs: http://ipython.readthedocs.io/en/stable/install/kernel_install.html


Instruction:
SQLAlchemy "default" vs "server_default" performance

Is there a performance advantage (or disadvantage) when using default instead of server_default for mapping table column default values when using SQLAlchemy with PostgreSQL? My understanding is that default renders the expression in the INSERT (usually) and that server_default places the expression in the CREATE TABLE statement. Seems like server_default is analogous to typical handling of defaults directly in the db such as: CREATE TABLE example ( id serial PRIMARY KEY, updated timestamptz DEFAULT now() ); ...but it is not clear to me if it is more efficient to handle defaults on INSERT or via table creation. Would there be any performance improvement or degradation for row inserts if each of the default parameters in the example below were changed to server_default? from uuid import uuid4 from sqlalchemy import Column, Boolean, DateTime, Integer from sqlalchemy.dialects.postgresql import UUID from sqlalchemy.ext.declarative import declarative_base from sqlalchemy.sql import func Base = declarative_base() class Item(Base): __tablename__ = 'item' id = Column(UUID(as_uuid=True), primary_key=True, default=uuid4) count = Column(Integer, nullable=False, default=0) flag = Column(Boolean, nullable=False, default=False) updated = Column(DateTime(timezone=True), nullable=False, default=func.now()) NOTE: The best explanation I found so far for when to use default instead of server_default does not address performance (see Mike Bayer's SO answer on the subject). My oversimplified summary of that explanation is that default is preferred over server_default when... The db can't handle the expression you need or want to use for the default value. You can't or don't want to modify the schema directly. ...so the question remains as to whether performance should be considered when choosing between default and server_default?

Output: It is impossible to give you a 'this is faster' answer, because performance per default value expression can vary widely, both on the server and in Python. A function to retrieve the current time behaves differently from a scalar default value. Next, you must realise that defaults can be provided in five different ways: Client-side scalar defaults. A fixed value, such a 0 or True. The value is used in an INSERT statement. Client-side Python function. Called each time a default is needed, produces the value to insert, used the same way as a scalar default from there on out. These can be context sensitive (have access to the current execution context with values to be inserted). Client-side SQL expression; this generates an extra piece of SQL expression that is then used in the query and executed on the server to produce a value. Server-side DLL expression are SQL expressions that are then stored in the table definition, so are part of the schema. The server uses these to fill a value for any columns omitted from INSERT statements, or when a column value is set to DEFAULT in an INSERT or UPDATE statement. Server-side implicit defaults or triggers, where other DLL such as triggers or specific database features provide a default value for columns. Note that when it comes to a SQL expression determining the default value, be that a client-side SQL expression, a server-side DLL expression, or a trigger, it makes very little difference to a database where the default value expression is coming from. The query executor will need to know how to produce values for a given column, once that's parsed out of the DML statement or the schema definition, the server still has to execute the expression for each row. Choosing between these options is rarely going to be based on performance alone, performance should at most be but one of multiple aspects you consider. There are many factors involved here: default with a scalar or Python function directly produces a Python default value, then sends the new value to the server when inserting. Python code can access the default value before the data is inserted into the database. A client-side SQL expression, a server_default value, and server-side implicit defaults and triggers all have the server generate the default, which then must be fetched by the client if you want to be able to access it in the same SQLAlchemy session. You can't access the value until the object has been inserted into the database. Depending on the exact query and database support, SQLAlchemy may have to make extra SQL queries to either generate a default before the INSERT statement or run a separate SELECT afterwards to fetch the defaults that have been inserted. You can control when this happens (directly when inserting or on first access after flushing, with the eager_defaults mapper configuration). If you have multiple clients on different platforms accessing the same database, a server_default or other default attached to the schema (such as a trigger) ensures that all clients will use the same defaults, regardless, while defaults implemented in Python can't be accessed by other platforms. When using PostgreSQL, SQLAlchemy can make use of the RETURNING clause for DML statements, which gives a client access to server-side generated defaults in a single step. So when using a server_default column default that calculates a new value for each row (not a scalar value), you save a small amount of Python-side time, and save a small amount of network bandwidth as you are not sending data for that column over to the database. The database could be faster creating that same value, or it could be slower; it largely depends on the type of operation. If you need to have access to the generated default value from Python, in the same transaction, you do then have to wait for a return stream of data, parsed out by SQLAlchemy. All these details can become insignificant compared to everything else that happens around inserting or updating rows, however. Do understand that a ORM is not suitable to be used for high-performance bulk row inserts or updates; quoting from the SQAlchemy Performance FAQ entry: The SQLAlchemy ORM uses the unit of work pattern when synchronizing changes to the database. This pattern goes far beyond simple “inserts” of data. It includes that attributes which are assigned on objects are received using an attribute instrumentation system which tracks changes on objects as they are made, includes that all rows inserted are tracked in an identity map which has the effect that for each row SQLAlchemy must retrieve its “last inserted id” if not already given, and also involves that rows to be inserted are scanned and sorted for dependencies as needed. Objects are also subject to a fair degree of bookkeeping in order to keep all of this running, which for a very large number of rows at once can create an inordinate amount of time spent with large data structures, hence it’s best to chunk these. Basically, unit of work is a large degree of automation in order to automate the task of persisting a complex object graph into a relational database with no explicit persistence code, and this automation has a price. ORMs are basically not intended for high-performance bulk inserts - this is the whole reason SQLAlchemy offers the Core in addition to the ORM as a first-class component. Because an ORM like SQLAlchemy comes with a hefty overhead price, any performance differences between a server-side or Python-side default quickly disappears in the noise of ORM operations. So if you are concerned about performance for large-quantity insert or update operations, you would want to use bulk operations for those, and enable the psycopg2 batch execution helpers to really get a speed boost. When using these bulk operations, I'd expect server-side defaults to improve performance just by saving bandwidth moving row data from Python to the server, but how much depends on the exact nature of the default values. If ORM insert and update performance outside of bulk operations is a big issue for you, you need to test your specific options. I'd start with the SQLAlchemy examples.performance package and add your own test suite using two models that differ only in a single server_default and default configuration.


Instruction:
How to set the root directory for Visual Studio Code Python Extension?

I have no trouble running and debugging my project with VSCode Python Extension (ms-python.python), but since python sub-project root directory is not the whole project directory, all imports from my sources are underlined with red color and are listed in the problems and so Go to definition and some similar features don't work properly. How can I tell the IDE where's the start point of my project: Whole Project path: docs server entities user.py customer.py env viewer db The server directory is where the imports path are started from: from entities.user import User

Output: You can create a .env file with: PYTHONPATH=server That will add your server folder to PYTHONPATH as needed. (You may need to restart VSCode for it to take PYTHONPATH into account correctly.) Edited to clarify... Create a file named .env under the repo root e.g. your_repo/.env. Also creating the file under the folder where your consuming code is, instead of under repo root, seems to work e.g. your_repo/service/.env. For more details, see documentation on environment variable definition files. For me this worked without restarting VSC, perhaps this is a matter of newer VSC and extensions versions.


Instruction:
export notebook to pdf without code

I have a large notebook with a lot of figures and text. I want to convert it to a html file. However, I don't want to export the code. I am using the following command ipython nbconvert --to html notebook.ipynb But this option also exports the code. Is there a way to convert the notebook to html without the code?

Output: I found this article interesting it explains how to remove the input columns : you have to create a template file named "hidecode.tplx" in same directory as the notebook you want to convert and add those line in it : ((*- extends 'article.tplx' -*)) ((* block input_group *)) ((*- if cell.metadata.get('nbconvert', {}).get('show_code', False) -*)) ((( super() ))) ((*- endif -*)) ((* endblock input_group *)) And after run this command it will use pdfLatex to convert the notebook in pdf via latex: jupyter nbconvert --to pdf --template hidecode Example.ipynb or if you want to edit you can convert it to a .tex document and use pdfLatex to put it in pdf : jupyter nbconvert --to latex --template hidecode Example.ipynb EDIT Sept 2018: ipython nbconvert is deprecated. It will be replaced by jupyter nbconvert: So we replace the command ipython with jupyter EDIT Feb 2021: (This is my best answer here, so let me take care of it) Following @Mrule comment adding --no-input flag will make it work without the template... jupyter nbconvert --to latex --no-input Example.ipynb PS: If you are getting issue saying : LaTeX error related to tcolorbox.sty not found Please refer to this guide to update your tex installation and this question


Instruction:
How can I set two primary key fields for my models in Django?

I have a model like this: class Hop(models.Model): migration = models.ForeignKey('Migration') host = models.ForeignKey(User, related_name='host_set') How can I have the primary key be the combination of migration and host?

Output: Update Django 4.0 Django 4.0 documentation recommends using UniqueConstraint with the constraints option instead of unique_together. Use UniqueConstraint with the constraints option instead. UniqueConstraint provides more functionality than unique_together. unique_together may be deprecated in the future. class Hop(models.Model): migration = models.ForeignKey('Migration') host = models.ForeignKey(User, related_name='host_set') class Meta: constraints = [ models.UniqueConstraint( fields=['migration', 'host'], name='unique_migration_host_combination' ) ] Original Answer I would implement this slightly differently. I would use a default primary key (auto field), and use the meta class property, unique_together class Hop(models.Model): migration = models.ForeignKey('Migration') host = models.ForeignKey(User, related_name='host_set') class Meta: unique_together = (("migration", "host"),) It would act as a "surrogate" primary key column. If you really want to create a multi-column primary key, look into this app


Instruction:
Check if input is a list/tuple of strings or a single string

I've a method that I want to be able to accept either a single string (a path, but not necessarily one that exists on the machine running the code) or a list/tuple of strings. Given that strings act as lists of characters, how can I tell which kind the method has received? I'd like to be able to accept either standard or unicode strings for a single entry, and either lists or tuples for multiple, so isinstance doesn't seem to be the answer unless I'm missing a clever trick with it (like taking advantage of common ancestor classes?). Python version is 2.5

Output: You can check if a variable is a string or unicode string with Python 3: isinstance(some_object, str) Python 2: isinstance(some_object, basestring) This will return True for both strings and unicode strings As you are using python 2.5, you could do something like this: if isinstance(some_object, basestring): ... elif all(isinstance(item, basestring) for item in some_object): # check iterable for stringness of all items. Will raise TypeError if some_object is not iterable ... else: raise TypeError # or something along that line Stringness is probably not a word, but I hope you get the idea


Instruction:
Conda environments and .BAT files

I am setting up calls to python (Anaconda distribution) via BAT files and the windows task scheduler. I've now used environments for the first time and was trying to set a .bat file up as below: activate [my_env] python my_script.py deactivate Unfortunately it appears that the second command does not get executed.

Output: Use the 'call' command when activating/deactivating the environment. call activate [my_env] python my_script.py call conda deactivate See https://github.com/conda/conda/issues/794


Instruction:
What is the meaning of curly braces?

Just starting to figure Python out. I've read this question and its responses: Is it true that I can't use curly braces in Python? and I still can't fathom how curly braces work, especially since pages like Simple Programs: http://wiki.python.org/moin/SimplePrograms use curly braces all over the place. I understand square brackets and regular curved parentheses, but I don't know what's meant by "defining dictionaries" or what they're supposed to represent.

Output: "Curly Braces" are used in Python to define a dictionary. A dictionary is a data structure that maps one value to another - kind of like how an English dictionary maps a word to its definition. Python: dict = { "a" : "Apple", "b" : "Banana", } They are also used to format strings, instead of the old C style using %, like: ds = ['a', 'b', 'c', 'd'] x = ['has_{} 1'.format(d) for d in ds] print x ['has_a 1', 'has_b 1', 'has_c 1', 'has_d 1'] They are not used to denote code blocks as they are in many "C-like" languages. C: if (condition) { // do this } Update: In addition to Python's dict data types Python has (since Python 2.7) set as well, which uses curly braces too and are declared as follows: my_set = {1, 2, 3, 4}


Instruction:
How to return history of validation loss in Keras

Using Anaconda Python 2.7 Windows 10. I am training a language model using the Keras exmaple: print('Build model...') model = Sequential() model.add(GRU(512, return_sequences=True, input_shape=(maxlen, len(chars)))) model.add(Dropout(0.2)) model.add(GRU(512, return_sequences=False)) model.add(Dropout(0.2)) model.add(Dense(len(chars))) model.add(Activation('softmax')) model.compile(loss='categorical_crossentropy', optimizer='rmsprop') def sample(a, temperature=1.0): # helper function to sample an index from a probability array a = np.log(a) / temperature a = np.exp(a) / np.sum(np.exp(a)) return np.argmax(np.random.multinomial(1, a, 1)) # train the model, output generated text after each iteration for iteration in range(1, 3): print() print('-' * 50) print('Iteration', iteration) model.fit(X, y, batch_size=128, nb_epoch=1) start_index = random.randint(0, len(text) - maxlen - 1) for diversity in [0.2, 0.5, 1.0, 1.2]: print() print('----- diversity:', diversity) generated = '' sentence = text[start_index: start_index + maxlen] generated += sentence print('----- Generating with seed: "' + sentence + '"') sys.stdout.write(generated) for i in range(400): x = np.zeros((1, maxlen, len(chars))) for t, char in enumerate(sentence): x[0, t, char_indices[char]] = 1. preds = model.predict(x, verbose=0)[0] next_index = sample(preds, diversity) next_char = indices_char[next_index] generated += next_char sentence = sentence[1:] + next_char sys.stdout.write(next_char) sys.stdout.flush() print() According to Keras documentation, the model.fit method returns a History callback, which has a history attribute containing the lists of successive losses and other metrics. hist = model.fit(X, y, validation_split=0.2) print(hist.history) After training my model, if I run print(model.history) I get the error: AttributeError: 'Sequential' object has no attribute 'history' How do I return my model history after training my model with the above code? UPDATE The issue was that: The following had to first be defined: from keras.callbacks import History history = History() The callbacks option had to be called model.fit(X_train, Y_train, nb_epoch=5, batch_size=16, callbacks=[history]) But now if I print print(history.History) it returns {} even though I ran an iteration.

Output: It's been solved. The losses only save to the History over the epochs. I was running iterations instead of using the Keras built in epochs option. so instead of doing 4 iterations I now have model.fit(......, nb_epoch = 4) Now it returns the loss for each epoch run: print(hist.history) {'loss': [1.4358016599558268, 1.399221191623641, 1.381293383180471, 1.3758836857303727]}


Instruction:
"import datetime" v.s. "from datetime import datetime"

I have a script that needs to execute the following at different lines in the script: today_date = datetime.date.today() date_time = datetime.strp(date_time_string, '%Y-%m-%d %H:%M') In my import statements I have the following: from datetime import datetime import datetime I get the following error: AttributeError: 'module' object has no attribute 'strp' If I change the order of the import statements to: import datetime from datetime import datetime I get the following error: AttributeError: 'method_descriptor' object has no attribute 'today' If I again change the import statement to: import datetime I get the following error: AttributeError: 'module' object has no attribute 'strp' What is going on here and how do I get both to work?

Output: Your trouble is that you have some code that is expecting datetime to be a reference to the datetime module and other code that is expecting datetime to be a reference to the datetime class. Obviously, it can't be both. When you do: from datetime import datetime import datetime You are first setting datetime to be a reference to the class, then immediately setting it to be a reference to the module. When you do it the other way around, it instead ends up being a reference to the class. Last assignment "wins." You need to rename one of these references. For example: import datetime as dt from datetime import datetime Then you can change references in the form datetime.xxxx that refer to the module to dt.xxxx. Or else just import datetime and change all references to use the module name. In other words, if something just says datetime(...) you need to change that reference to datetime.datetime. Python has a fair bit of this kind of thing in its library, unfortunately. If they followed their own naming guidelines in PEP 8, the datetime class would be named Datetime and there'd be no problem using both datetime to mean the module and Datetime to mean the class.


Instruction:
check if a file is open in Python

In my app, I write to an excel file. After writing, the user is able to view the file by opening it. But if the user forgets to close the file before any further writing, a warning message should appear. So I need a way to check this file is open before the writing process. Could you supply me with some python code to do this task?

Output: I assume that you're writing to the file, then closing it (so the user can open it in Excel), and then, before re-opening it for append/write operations, you want to check that the file isn't still open in Excel? This is how you could do that: while True: # repeat until the try statement succeeds try: myfile = open("myfile.csv", "r+") # or "a+", whatever you need break # exit the loop except IOError: input("Could not open file! Please close Excel. Press Enter to retry.") # restart the loop with myfile: do_stuff()


Instruction:
How can I migrate from Poetry to UV package manager?

I'm planning to switch from poetry to the uv Python package manager, but I can't find any migration guides. Currently, I'm using Poetry and already have a pyproject.toml file. What key(s) should be modified or added to migrate properly to uv? Here’s the current pyproject.toml structure: [tool.poetry] name = "name" version = "1.6.0" description = "" authors = [ "...", ] maintainers = [ "...", ] readme = "README.md" [tool.poetry.dependencies] python = "^3.12" fastapi = "^0.115.2" uvicorn = { version = "^0.32.0", extras = ["standard"] } pydantic = "^2.5.3" pydantic-settings = "^2" [tool.poetry.group.dev.dependencies] pytest = "^8.3.3" flake8 = "~7.1.1" mypy = "^1.12" [tool.isort] profile = "black" multi_line_output = 3 [tool.mypy] strict = true ignore_missing_imports = true [tool.pytest.ini_options] filterwarnings = [ "error", "ignore::DeprecationWarning", "ignore:.*unclosed.*:ResourceWarning", ] env = [ "...=...", "...=...", ] [tool.coverage.run] omit = [ "...=...", "...=...", ] [tool.coverage.report] exclude_lines = [ "pragma: no cover", "if TYPE_CHECKING", ] [build-system] requires = ["poetry-core>=1.0.0"] build-backend = "poetry.core.masonry.api" Additionally, in the [build-system] section, I currently have poetry-core. Should this be replaced with something specific for uv during the migration?

Output: Another very recently released tool is migrate-to-uv. Run: uvx migrate-to-uv to migrate from Poetry or pipenv to uv. This will rewrite your pyproject.toml file and remove poetry.lock. You still need to uv lock afterwards.


Instruction:
Django orm get latest for each group

I am using Django 1.6 with Mysql. I have these models: class Student(models.Model): username = models.CharField(max_length=200, unique = True) class Score(models.Model): student = models.ForeignKey(Student) date = models.DateTimeField() score = models.IntegerField() I want to get the latest score record for each student. I have tried: Score.objects.values('student').annotate(latest_date=Max('date')) and: Score.objects.values('student__username').annotate(latest_date=Max('date')) as described Django ORM - Get the latest record for the group but it did not help.

Output: This should work on Django 1.2+ and MySQL: Score.objects.annotate( max_date=Max('student__score__date') ).filter( date=F('max_date') )


Instruction:
Summing the contents of two collections.Counter() objects

I am working with collections.Counter() counters. I would like to combine two of them in a meaningful manner. Suppose I have 2 counters, say, Counter({'menu': 20, 'good': 15, 'happy': 10, 'bar': 5}) and Counter({'menu': 1, 'good': 1, 'bar': 3}) I am trying to end up with: Counter({'menu': 21, 'good': 16, 'happy': 10,'bar': 8}) How can I do this?

Output: All you need to do is add them: >>> from collections import Counter >>> a = Counter({'menu': 20, 'good': 15, 'happy': 10, 'bar': 5}) >>> b = Counter({'menu': 1, 'good': 1, 'bar': 3}) >>> a + b Counter({'menu': 21, 'good': 16, 'happy': 10, 'bar': 8}) From the docs: Several mathematical operations are provided for combining Counter objects to produce multisets (counters that have counts greater than zero). Addition and subtraction combine counters by adding or subtracting the counts of corresponding elements. Note that if you want to save memory by modifying the Counter in-place rather than creating a new one, you can do a.update(b) or b.update(a).


Instruction:
Getting started with the Python debugger, pdb

I want to add pdb—the Python debugger—to my toolbox. What's the best way to get started?

Output: Here's a list of resources to get started with the Python debugger: Read Steve Ferb's article "Debugging in Python" Watch Eric Holscher's screencast "Using pdb, the Python Debugger" Read the Python documentation for pdb — The Python Debugger Read Chapter 9—When You Don't Even Know What to Log: Using Debuggers—of Karen Tracey's Django 1.1 Testing and Debugging.


Instruction:
Inserting new records with one-to-many relationship in sqlalchemy

I'm following the flask-sqlalchemy tutorial on declaring models regarding one-to-many relationship. The example code is as follows: class Person(db.Model): id = db.Column(db.Integer, primary_key=True) name = db.Column(db.String(50)) addresses = db.relationship('Address', backref='person', lazy='dynamic') class Address(db.Model): id = db.Column(db.Integer, primary_key=True) email = db.Column(db.String(50)) person_id = db.Column(db.Integer, db.ForeignKey('person.id')) Now I'm wondering how to insert new records into DB using such model. I assume I need a constructor init, but I have difficulties to understand how it should be implemented and used. The main problem for me here is that Person depends on Address and Address has ForeignKey to Person, so it should know about the Person in advance. Plase help me to understand how it should be performed. Thank you in advance.

Output: You dont need to write a constructor, you can either treat the addresses property on a Person instance as a list: a = Address(email='foo@bar.com') p = Person(name='foo') p.addresses.append(a) Or you can pass a list of addresses to the Person constructor a = Address(email='foo@bar.com') p = Person(name='foo', addresses=[a]) In either case you can then access the addresses on your Person instance like so: db.session.add(p) db.session.add(a) db.session.commit() print(p.addresses.count()) # 1 print(p.addresses[0]) # <Address object at 0x10c098ed0> print(p.addresses.filter_by(email='foo@bar.com').count()) # 1


Instruction:
Which maximum does Python pick in the case of a tie?

When using the max() function in Python to find the maximum value in a list (or tuple, dict etc.) and there is a tie for maximum value, which one does Python pick? Is it random? This is relevant if, for instance, one has a list of tuples and one selects a maximum (using a key=) based on the first element of the tuple but there are different second elements. How does Python decide which one to pick as the maximum?

Output: It picks the first element it sees. See the documentation for max(): If multiple items are maximal, the function returns the first one encountered. This is consistent with other sort-stability preserving tools such as sorted(iterable, key=keyfunc, reverse=True)[0] and heapq.nlargest(1, iterable, key=keyfunc). In the source code this is implemented in ./Python/bltinmodule.c by builtin_max, which wraps the more general min_max function. min_max will iterate through the values and use PyObject_RichCompareBool to see if they are greater than the current value. If so, the greater value replaces it. Equal values will be skipped over. The result is that the first maximum will be chosen in the case of a tie.


Instruction:
How to get the format of image with PIL?

After loading an image file with PIL.Image, how can I determine whether the image file is a PNG/JPG/BMP/GIF? I understand very little about these file formats, can PIL get the format metadata from the file header? Or does it need to 'analyze' the data within the file? If PIL doesn't provide such an API, is there any python library that does?

Output: Try: from PIL import Image img = Image.open(filename) print(img.format) # 'JPEG' More info https://pillow.readthedocs.io/en/latest/reference/Image.html#PIL.Image.Image.format https://pillow.readthedocs.io/en/latest/handbook/image-file-formats.html


Instruction:
Renaming file extension using pathlib (python 3)

I am using windows 10 and winpython. I have a file with a .dwt extension (it is a text file). I want to change the extension of this file to .txt. My code does not throw any errors, but it does not change the extension. from pathlib import Path filename = Path("E:\\seaborn_plot\\x.dwt") print(filename) filename_replace_ext = filename.with_suffix('.txt') print(filename_replace_ext) Expected results are printed out (as shown below) in winpython's ipython window output: E:\seaborn_plot\x.dwt E:\seaborn_plot\x.txt But when I look for a file with a renamed extension, the extension has not been changed, only the original file exists. I suspect windows file permissions.

Output: You have to actually rename the file not just print out the new name. Use Path.rename() from pathlib import Path my_file = Path("E:\\seaborn_plot\\x.dwt") my_file.rename(my_file.with_suffix('.txt')) Note: To replace the target if it exists use Path.replace() Use os.rename() import os my_file = 'E:\\seaborn_plot\\x.dwt' new_ext = '.txt' # Gets my_file minus the extension name_without_ext = os.path.splitext(my_file)[0] os.rename(my_file, name_without_ext + new_ext) Ref: os.path.splitext(path) PurePath.with_suffix(suffix)


Instruction:
How can I get the current time (now) in UTC?

I have a python datetime object (representing five minutes from now) which I would like to convert to UTC. I am planning to output it in RFC 2822 format to put in an HTTP header, but I am not sure if that matters for this question. I found some information on this site about converting time objects, and it looks simpler that way, but this time I really want to use datetime objects, because I am using timedeltas to adjust them: I tried something like this: from datetime import datetime, timedelta now = datetime.now() fiveMinutesLater = datetime.now() + timedelta(minutes=5) fiveMinutesLaterUtc = ??? Nothing in the time or datetime module looks like it would help me. It seems like I may be able to do it by passing the datetime object through 3 or 4 functions, but I am wondering if there is a simpler way. I would prefer not to use third-party modules, but I may if it is the only reasonable choice.

Output: Run this to obtain a naive datetime in UTC (and to add five minutes to it): >>> from datetime import datetime, timedelta >>> datetime.utcnow() datetime.datetime(2021, 1, 26, 15, 41, 52, 441598) >>> datetime.utcnow() + timedelta(minutes=5) datetime.datetime(2021, 1, 26, 15, 46, 52, 441598) If you would prefer a timezone-aware datetime object, run this in Python 3.2 or higher: >>> from datetime import datetime, timezone >>> datetime.now(timezone.utc) datetime.datetime(2021, 1, 26, 15, 43, 54, 379421, tzinfo=datetime.timezone.utc)


Instruction:
Iterate over individual bytes in Python 3

When iterating over a bytes object in Python 3, one gets the individual bytes as ints: >>> [b for b in b'123'] [49, 50, 51] How to get 1-length bytes objects instead? The following is possible, but not very obvious for the reader and most likely performs bad: >>> [bytes([b]) for b in b'123'] [b'1', b'2', b'3']

Output: int.to_bytes int objects have a to_bytes method which can be used to convert an int to its corresponding byte: >>> import sys >>> [i.to_bytes(1, sys.byteorder) for i in b'123'] [b'1', b'2', b'3'] As with some other other answers, it's not clear that this is more readable than the OP's original solution: the length and byteorder arguments make it noisier I think. struct.unpack Another approach would be to use struct.unpack, though this might also be considered difficult to read, unless you are familiar with the struct module: >>> import struct >>> struct.unpack('3c', b'123') (b'1', b'2', b'3') (As jfs observes in the comments, the format string for struct.unpack can be constructed dynamically; in this case we know the number of individual bytes in the result must equal the number of bytes in the original bytestring, so struct.unpack(str(len(bytestring)) + 'c', bytestring) is possible.) Performance >>> import random, timeit >>> bs = bytes(random.randint(0, 255) for i in range(100)) >>> # OP's solution >>> timeit.timeit(setup="from __main__ import bs", stmt="[bytes([b]) for b in bs]") 46.49886950897053 >>> # Accepted answer from jfs >>> timeit.timeit(setup="from __main__ import bs", stmt="[bs[i:i+1] for i in range(len(bs))]") 20.91463226894848 >>> # Leon's answer >>> timeit.timeit(setup="from __main__ import bs", stmt="list(map(bytes, zip(bs)))") 27.476876026019454 >>> # guettli's answer >>> timeit.timeit(setup="from __main__ import iter_bytes, bs", stmt="list(iter_bytes(bs))") 24.107485140906647 >>> # user38's answer (with Leon's suggested fix) >>> timeit.timeit(setup="from __main__ import bs", stmt="[chr(i).encode('latin-1') for i in bs]") 45.937552741961554 >>> # Using int.to_bytes >>> timeit.timeit(setup="from __main__ import bs;from sys import byteorder", stmt="[x.to_bytes(1, byteorder) for x in bs]") 32.197659170022234 >>> # Using struct.unpack, converting the resulting tuple to list >>> # to be fair to other methods >>> timeit.timeit(setup="from __main__ import bs;from struct import unpack", stmt="list(unpack('100c', bs))") 1.902243083808571 struct.unpack seems to be at least an order of magnitude faster than other methods, presumably because it operates at the byte level. int.to_bytes, on the other hand, performs worse than most of the "obvious" approaches.


Instruction:
Why does b+=(4,) work and b = b + (4,) doesn't work when b is a list?

If we take b = [1,2,3] and if we try doing: b+=(4,) It returns b = [1,2,3,4], but if we try doing b = b + (4,) it doesn't work. b = [1,2,3] b+=(4,) # Prints out b = [1,2,3,4] b = b + (4,) # Gives an error saying you can't add tuples and lists I expected b+=(4,) to fail as you can't add a list and a tuple, but it worked. So I tried b = b + (4,) expecting to get the same result, but it didn't work.

Output: The problem with "why" questions is that usually they can mean multiple different things. I will try to answer each one I think you might have in mind. "Why is it possible for it to work differently?" which is answered by e.g. this. Basically, += tries to use different methods of the object: __iadd__ (which is only checked on the left-hand side), vs __add__ and __radd__ ("reverse add", checked on the right-hand side if the left-hand side doesn't have __add__) for +. "What exactly does each version do?" In short, the list.__iadd__ method does the same thing as list.extend (but because of the language design, there is still an assignment back). This also means for example that >>> a = [1,2,3] >>> b = a >>> a += [4] # uses the .extend logic, so it is still the same object >>> b # therefore a and b are still the same list, and b has the `4` added [1, 2, 3, 4] >>> b = b + [5] # makes a new list and assigns back to b >>> a # so now a is a separate list and does not have the `5` [1, 2, 3, 4] +, of course, creates a new object, but explicitly requires another list instead of trying to pull elements out of a different sequence. "Why is it useful for += to do this? It's more efficient; the extend method doesn't have to create a new object. Of course, this has some surprising effects sometimes (like above), and generally Python is not really about efficiency, but these decisions were made a long time ago. "What is the reason not to allow adding lists and tuples with +?" See here (thanks, @splash58); one idea is that (tuple + list) should produce the same type as (list + tuple), and it's not clear which type the result should be. += doesn't have this problem, because a += b obviously should not change the type of a.


Instruction:
anaconda - path environment variable in windows

I'm trying to run Python from the Windows command prompt (windows 10). So the result is the typical one when the path environment variable is not configured c:\windows\system32>python 'python' is not recognized as an internal or external command, operable program or batch file However, I'm unsure which directory I should set up in the path variable. I tried a few variations, and none of them work, including: c:\users\xxx\anaconda3 c:\users\xxx\anaconda3\Scripts c:\users\xxx\anaconda3\libs\python34 and none of them works. Does anyone have experience with this system constellation (windows, anaconda)? Thanks.

Output: It turns out I was mistaken. Here is the solution: In anaconda (as well as in other implementations), set the path environment variable to the directory where 'python.exe' is installed. As a default, the python.exe file in anaconda is in: c:\.....\anaconda After you do that, the Python command works, in my case, yielding the following: python Python 3.4.3 |Anaconda 2.2.0. (64|bit)|(default, Nov 7 2015), etc, etc


Instruction:
Interactive matplotlib plot with two sliders

I used matplotlib to create some plot, which depends on 8 variables. I would like to study how the plot changes when I change some of them. I created some script that calls the matplotlib one and generates different snapshots that later I convert into a movie, it is not bad, but a bit clumsy. I wonder if somehow I could interact with the plot regeneration using keyboard keys to increase / decrease values of some of the variables and see instantly how the plot changes. What is the best approach for this? Also if you can point me to interesting links or a link with a plot example with just two sliders?

Output: In addition to what @triplepoint mentioned, have a look at the slider widget. There's an example on the matplotlib examples page. It's a graphical slider bar rather than keyboard bindings, but it works quite well for what you want to do. Also note that to guarantee the sliders and buttons remain responsive and not garbage-collected, references to the objects (amp_slider, freq_slider, etc.) should be maintained by yourself. (I'm making this community wiki, as I'm just copy-pasting from the example. This particular example teaches bad habits (e.g. from pylab import *), but it gets the point across. The example has been fixed to avoid the use of pylab.) from numpy import pi, sin import numpy as np import matplotlib.pyplot as plt from matplotlib.widgets import Slider, Button, RadioButtons def signal(amp, freq): return amp * sin(2 * pi * freq * t) axis_color = 'lightgoldenrodyellow' fig = plt.figure() ax = fig.add_subplot(111) # Adjust the subplots region to leave some space for the sliders and buttons fig.subplots_adjust(left=0.25, bottom=0.25) t = np.arange(0.0, 1.0, 0.001) amp_0 = 5 freq_0 = 3 # Draw the initial plot # The 'line' variable is used for modifying the line later [line] = ax.plot(t, signal(amp_0, freq_0), linewidth=2, color='red') ax.set_xlim([0, 1]) ax.set_ylim([-10, 10]) # Add two sliders for tweaking the parameters # Define an axes area and draw a slider in it amp_slider_ax = fig.add_axes([0.25, 0.15, 0.65, 0.03], facecolor=axis_color) amp_slider = Slider(amp_slider_ax, 'Amp', 0.1, 10.0, valinit=amp_0) # Draw another slider freq_slider_ax = fig.add_axes([0.25, 0.1, 0.65, 0.03], facecolor=axis_color) freq_slider = Slider(freq_slider_ax, 'Freq', 0.1, 30.0, valinit=freq_0) # Define an action for modifying the line when any slider's value changes def sliders_on_changed(val): line.set_ydata(signal(amp_slider.val, freq_slider.val)) fig.canvas.draw_idle() amp_slider.on_changed(sliders_on_changed) freq_slider.on_changed(sliders_on_changed) # Add a button for resetting the parameters reset_button_ax = fig.add_axes([0.8, 0.025, 0.1, 0.04]) reset_button = Button(reset_button_ax, 'Reset', color=axis_color, hovercolor='0.975') def reset_button_on_clicked(mouse_event): freq_slider.reset() amp_slider.reset() reset_button.on_clicked(reset_button_on_clicked) # Add a set of radio buttons for changing color color_radios_ax = fig.add_axes([0.025, 0.5, 0.15, 0.15], facecolor=axis_color) color_radios = RadioButtons(color_radios_ax, ('red', 'blue', 'green'), active=0) def color_radios_on_clicked(label): line.set_color(label) fig.canvas.draw_idle() color_radios.on_clicked(color_radios_on_clicked) plt.show()


Instruction:
Why am I getting "invalid syntax" from an f-string?

I cannot get f-strings to work in Python 3. I tried this at the REPL: In [1]: state = "Washington" In [2]: state Out[2]: 'Washington' In [3]: my_message = f"I live in {state}" File "<ipython-input-3-d004dd9e0255>", line 1 my_message = f"I live in {state}" ^ SyntaxError: invalid syntax I figured my machine was defaulting to python 2, but a quick check reveals: Python 3.5.2 (default, Nov 17 2016, 17:05:23) Type "copyright", "credits" or "license" for more information. IPython 5.2.2 -- An enhanced Interactive Python. What is wrong, and how do I fix it?

Output: As suggested by Josh Lee in the comment section, that kind of string interpolation was added in Python 3.6 only, see What’s New In Python 3.6 (here it's called "PEP 498: Formatted string literals"). However, you seem to be using Python 3.5.2, which does not support that syntax.


Instruction:
Why does Python allow out-of-range slice indexes for sequences?

So I just came across what seems to me like a strange Python feature and wanted some clarification about it. The following array manipulation somewhat makes sense: p = [1,2,3] p[3:] = [4] p = [1,2,3,4] I imagine it is actually just appending this value to the end, correct? Why can I do this, however? p[20:22] = [5,6] p = [1,2,3,4,5,6] And even more so this: p[20:100] = [7,8] p = [1,2,3,4,5,6,7,8] This just seems like wrong logic. It seems like this should throw an error! Any explanation? -Is it just a weird thing Python does? -Is there a purpose to it? -Or am I thinking about this the wrong way?

Output: Part of question regarding out-of-range indices Slice logic automatically clips the indices to the length of the sequence. Allowing slice indices to extend past end points was done for convenience. It would be a pain to have to range check every expression and then adjust the limits manually, so Python does it for you. Consider the use case of wanting to display no more than the first 50 characters of a text message. The easy way (what Python does now): preview = msg[:50] Or the hard way (do the limit checks yourself): n = len(msg) preview = msg[:50] if n > 50 else msg Manually implementing that logic for adjustment of end points would be easy to forget, would be easy to get wrong (updating the 50 in two places), would be wordy, and would be slow. Python moves that logic to its internals where it is succint, automatic, fast, and correct. This is one of the reasons I love Python :-) Part of question regarding assignments length mismatch from input length The OP also wanted to know the rationale for allowing assignments such as p[20:100] = [7,8] where the assignment target has a different length (80) than the replacement data length (2). It's easiest to see the motivation by an analogy with strings. Consider, "five little monkeys".replace("little", "humongous"). Note that the target "little" has only six letters and "humongous" has nine. We can do the same with lists: >>> s = list("five little monkeys") >>> i = s.index('l') >>> n = len('little') >>> s[i : i+n ] = list("humongous") >>> ''.join(s) 'five humongous monkeys' This all comes down to convenience. Prior to the introduction of the copy() and clear() methods, these used to be popular idioms: s[:] = [] # clear a list t = u[:] # copy a list Even now, we use this to update lists when filtering: s[:] = [x for x in s if not math.isnan(x)] # filter-out NaN values Hope these practical examples give a good perspective on why slicing works as it does.


Instruction:
Git - Should Pipfile.lock be committed to version control?

When two developers are working on a project with different operating systems, the Pipfile.lock is different (especially the part inside host-environment-markers). For PHP, most people recommend to commit composer.lock file. Do we have to do the same for Python?

Output: Short - Yes! The lock file tells pipenv exactly which version of each dependency needs to be installed. You will have consistency across all machines. // update: Same question on github


Instruction:
How do I convert an array (i.e. list) column to Vector

Short version of the question! Consider the following snippet (assuming spark is already set to some SparkSession): from pyspark.sql import Row source_data = [ Row(city="Chicago", temperatures=[-1.0, -2.0, -3.0]), Row(city="New York", temperatures=[-7.0, -7.0, -5.0]), ] df = spark.createDataFrame(source_data) Notice that the temperatures field is a list of floats. I would like to convert these lists of floats to the MLlib type Vector, and I'd like this conversion to be expressed using the basic DataFrame API rather than going via RDDs (which is inefficient because it sends all data from the JVM to Python, the processing is done in Python, we don't get the benefits of Spark's Catalyst optimizer, yada yada). How do I do this? Specifically: Is there a way to get a straight cast working? See below for details (and a failed attempt at a workaround)? Or, is there any other operation that has the effect I was after? Which is more efficient out of the two alternative solutions I suggest below (UDF vs exploding/reassembling the items in the list)? Or are there any other almost-but-not-quite-right alternatives that are better than either of them? A straight cast doesn't work This is what I would expect to be the "proper" solution. I want to convert the type of a column from one type to another, so I should use a cast. As a bit of context, let me remind you of the normal way to cast it to another type: from pyspark.sql import types df_with_strings = df.select( df["city"], df["temperatures"].cast(types.ArrayType(types.StringType()))), ) Now e.g. df_with_strings.collect()[0]["temperatures"][1] is '-7.0'. But if I cast to an ml Vector then things do not go so well: from pyspark.ml.linalg import VectorUDT df_with_vectors = df.select(df["city"], df["temperatures"].cast(VectorUDT())) This gives an error: pyspark.sql.utils.AnalysisException: "cannot resolve 'CAST(`temperatures` AS STRUCT<`type`: TINYINT, `size`: INT, `indices`: ARRAY<INT>, `values`: ARRAY<DOUBLE>>)' due to data type mismatch: cannot cast ArrayType(DoubleType,true) to org.apache.spark.ml.linalg.VectorUDT@3bfc3ba7;; 'Project [city#0, unresolvedalias(cast(temperatures#1 as vector), None)] +- LogicalRDD [city#0, temperatures#1] " Yikes! Any ideas how to fix this? Possible alternatives Alternative 1: Using VectorAssembler There is a Transformer that seems almost ideal for this job: the VectorAssembler. It takes one or more columns and concatenates them into a single vector. Unfortunately it only takes Vector and Float columns, not Array columns, so the follow doesn't work: from pyspark.ml.feature import VectorAssembler assembler = VectorAssembler(inputCols=["temperatures"], outputCol="temperature_vector") df_fail = assembler.transform(df) It gives this error: pyspark.sql.utils.IllegalArgumentException: 'Data type ArrayType(DoubleType,true) is not supported.' The best work around I can think of is to explode the list into multiple columns and then use the VectorAssembler to collect them all back up again: from pyspark.ml.feature import VectorAssembler TEMPERATURE_COUNT = 3 assembler_exploded = VectorAssembler( inputCols=["temperatures[{}]".format(i) for i in range(TEMPERATURE_COUNT)], outputCol="temperature_vector" ) df_exploded = df.select( df["city"], *[df["temperatures"][i] for i in range(TEMPERATURE_COUNT)] ) converted_df = assembler_exploded.transform(df_exploded) final_df = converted_df.select("city", "temperature_vector") This seems like it would be ideal, except that TEMPERATURE_COUNT be more than 100, and sometimes more than 1000. (Another problem is that the code would be more complicated if you don't know the size of the array in advance, although that is not the case for my data.) Does Spark actually generate an intermediate data set with that many columns, or does it just consider this an intermediate step that individual items pass through transiently (or indeed does it optimise this away step entirely when it sees that the only use of these columns is to be assembled into a vector)? Alternative 2: use a UDF A rather simpler alternative is to use a UDF to do the conversion. This lets me express quite directly what I want to do in one line of code, and doesn't require making a data set with a crazy number of columns. But all that data has to be exchanged between Python and the JVM, and every individual number has to be handled by Python (which is notoriously slow for iterating over individual data items). Here is how that looks: from pyspark.ml.linalg import Vectors, VectorUDT from pyspark.sql.functions import udf list_to_vector_udf = udf(lambda l: Vectors.dense(l), VectorUDT()) df_with_vectors = df.select( df["city"], list_to_vector_udf(df["temperatures"]).alias("temperatures") ) Ignorable remarks The remaining sections of this rambling question are some extra things I came up with while trying to find an answer. They can probably be skipped by most people reading this. Not a solution: use Vector to begin with In this trivial example it's possible to create the data using the vector type to begin with, but of course my data isn't really a Python list that I'm parallelizing, but instead is being read from a data source. But for the record, here is how that would look: from pyspark.ml.linalg import Vectors from pyspark.sql import Row source_data = [ Row(city="Chicago", temperatures=Vectors.dense([-1.0, -2.0, -3.0])), Row(city="New York", temperatures=Vectors.dense([-7.0, -7.0, -5.0])), ] df = spark.createDataFrame(source_data) Inefficient solution: use map() One possibility is to use the RDD map() method to transform the list to a Vector. This is similar to the UDF idea, except that its even worse because the cost of serialisation etc. is incurred for all the fields in each row, not just the one being operated on. For the record, here's what that solution would look like: df_with_vectors = df.rdd.map(lambda row: Row( city=row["city"], temperatures=Vectors.dense(row["temperatures"]) )).toDF() Failed attempt at a workaround for cast In desperation, I noticed that Vector is represented internally by a struct with four fields, but using a traditional cast from that type of struct doesn't work either. Here is an illustration (where I built the struct using a udf but the udf isn't the important part): from pyspark.ml.linalg import Vectors, VectorUDT from pyspark.sql.functions import udf list_to_almost_vector_udf = udf(lambda l: (1, None, None, l), VectorUDT.sqlType()) df_almost_vector = df.select( df["city"], list_to_almost_vector_udf(df["temperatures"]).alias("temperatures") ) df_with_vectors = df_almost_vector.select( df_almost_vector["city"], df_almost_vector["temperatures"].cast(VectorUDT()) ) This gives the error: pyspark.sql.utils.AnalysisException: "cannot resolve 'CAST(`temperatures` AS STRUCT<`type`: TINYINT, `size`: INT, `indices`: ARRAY<INT>, `values`: ARRAY<DOUBLE>>)' due to data type mismatch: cannot cast StructType(StructField(type,ByteType,false), StructField(size,IntegerType,true), StructField(indices,ArrayType(IntegerType,false),true), StructField(values,ArrayType(DoubleType,false),true)) to org.apache.spark.ml.linalg.VectorUDT@3bfc3ba7;; 'Project [city#0, unresolvedalias(cast(temperatures#5 as vector), None)] +- Project [city#0, <lambda>(temperatures#1) AS temperatures#5] +- LogicalRDD [city#0, temperatures#1] "

Output: For pyspark>=3.1.0 Since 3.1.0 there is a build-it solution: array_to_vector. Given your case: from pyspark.ml.functions import vector_to_array df = df.withColumn("temperatures_vectorized", vector_to_array("temperatures")) P.S: Also since 3.0.0 there is an opposite operation as well: vector_to_array


Instruction:
How to create a list of empty lists

Apologies if this has been answered before, but I couldn't find a similar question on here. I am pretty new to Python and what I am trying to create is as follows: list1 = [] list2 = [] results = [list1, list2] This code works absolutely fine, but I was wondering if there was a quicker way to do this in one line. I tried the following, which didn't work, but I hope it demonstrates the sort of thing that I'm after: result = [list1[], list2[]] Also, in terms of complexity, would having it on one line really make any difference? Or would it be three assignments in either case?

Output: For manually creating a specified number of lists, this would be good: empty_list = [ [], [], ..... ] In case, you want to generate a bigger number of lists, then putting it inside a for loop would be good: empty_lists = [ [] for _ in range(n) ]


Instruction:
"E: Unable to locate package python-pip" on Ubuntu 18.04

I am trying to install virtualenv on Ubuntu. First it said command 'pip' not found, so I typed sudo apt install python-pip then it said E: Unable to locate package python-pip I tried to reset WSL, download using cmd but it doesn't work with Ubuntu. I don't know why. Even though I have downloaded python3, virtualenv, and pip using cmd. It doesn't work with Ubuntu 18.04. It also fails on Ubuntu 14.04. aiki@LAPTOP-886AEJJG:~$ pip Command 'pip' not found, but can be installed with: sudo apt install python-pip aiki@LAPTOP-886AEJJG:~$ sudo apt install python-pip [sudo] password for aiki: Reading package lists... Done Building dependency tree Reading state information... Done E: Unable to locate package python-pip I'm trying to install jarvis and mycroft on win 10, but I need to use Ubuntu because it only works with Linux.

Output: Try following command sequence on Ubuntu terminal: sudo apt-get install software-properties-common sudo apt-add-repository universe sudo apt-get update sudo apt-get install python3-pip


Instruction:
Get .wav file length or duration

I'm looking for a way to find out the duration of a audio file (.wav) in python. So far i had a look at python wave library, mutagen, pymedia, pymad i was not able to get the duration of the wav file. Pymad gave me the duration but its not consistent.

Output: The duration is equal to the number of frames divided by the framerate (frames per second): import wave import contextlib fname = '/tmp/test.wav' with contextlib.closing(wave.open(fname,'r')) as f: frames = f.getnframes() rate = f.getframerate() duration = frames / float(rate) print(duration) Regarding @edwards' comment, here is some code to produce a 2-channel wave file: import math import wave import struct FILENAME = "/tmp/test.wav" freq = 440.0 data_size = 40000 frate = 1000.0 amp = 64000.0 nchannels = 2 sampwidth = 2 framerate = int(frate) nframes = data_size comptype = "NONE" compname = "not compressed" data = [(math.sin(2 * math.pi * freq * (x / frate)), math.cos(2 * math.pi * freq * (x / frate))) for x in range(data_size)] try: wav_file = wave.open(FILENAME, 'w') wav_file.setparams( (nchannels, sampwidth, framerate, nframes, comptype, compname)) for values in data: for v in values: wav_file.writeframes(struct.pack('h', int(v * amp / 2))) finally: wav_file.close() If you play the resultant file in an audio player, you'll find that is 40 seconds in duration. If you run the code above it also computes the duration to be 40 seconds. So I believe the number of frames is not influenced by the number of channels and the formula above is correct.


Instruction:
isPrime Function for Python Language

So I was able to solve this problem with a little bit of help from the internet and this is what I got: def isPrime(n): for i in range(2,int(n**0.5)+1): if n%i==0: return False return True But my question really is how to do it, but WHY. I understand that 1 is not considered a "prime" number even though it is, and I understand that if it divides by ANYTHING within the range it is automatically not a prime thus the return False statement. but my question is what role does the square-rooting the "n" play here? P.s. I am very inexperienced and have just been introduced to programming a month ago.

Output: Of many prime number tests floating around the Internet, consider the following Python function: def is_prime(n): if n == 2 or n == 3: return True if n < 2 or n%2 == 0: return False if n < 9: return True if n%3 == 0: return False r = int(n**0.5) # since all primes > 3 are of the form 6n ± 1 # start with f=5 (which is prime) # and test f, f+2 for being prime # then loop by 6. f = 5 while f <= r: print('\t',f) if n % f == 0: return False if n % (f+2) == 0: return False f += 6 return True Since all primes > 3 are of the form 6n ± 1, once we eliminate that n is: not 2 or 3 (which are prime) and not even (with n%2) and not divisible by 3 (with n%3) then we can test every 6th n ± 1. Consider the prime number 5003: print is_prime(5003) Prints: 5 11 17 23 29 35 41 47 53 59 65 True The line r = int(n**0.5) evaluates to 70 (the square root of 5003 is 70.7318881411 and int() truncates this value) Consider the next odd number (since all even numbers other than 2 are not prime) of 5005, same thing prints: 5 False The limit is the square root since x*y == y*x The function only has to go 1 loop to find that 5005 is divisible by 5 and therefore not prime. Since 5 X 1001 == 1001 X 5 (and both are 5005), we do not need to go all the way to 1001 in the loop to know what we know at 5! Now, let's look at the algorithm you have: def isPrime(n): for i in range(2, int(n**0.5)+1): if n % i == 0: return False return True There are two issues: It does not test if n is less than 2, and there are no primes less than 2; It tests every number between 2 and n**0.5 including all even and all odd numbers. Since every number greater than 2 that is divisible by 2 is not prime, we can speed it up a little by only testing odd numbers greater than 2. So: def isPrime2(n): if n==2 or n==3: return True if n%2==0 or n<2: return False for i in range(3, int(n**0.5)+1, 2): # only odd numbers if n%i==0: return False return True OK -- that speeds it up by about 30% (I benchmarked it...) The algorithm I used is_prime is about 2x times faster still, since only every 6th integer is looping through the loop. (Once again, I benchmarked it.) Side note: x**0.5 is the square root: >>> import math >>> math.sqrt(100)==100**0.5 True Side note 2: primality testing is an interesting problem in computer science.


Instruction:
Progress Bar while download file over http with Requests

I need to download a sizable (~200MB) file. I figured out how to download and save the file with here. It would be nice to have a progress bar to know how much has been downloaded. I found ProgressBar but I'm not sure how to incorporate the two together. Here's the code I tried, but it didn't work. bar = progressbar.ProgressBar(max_value=progressbar.UnknownLength) with closing(download_file()) as r: for i in range(20): bar.update(i)

Output: I suggest you try tqdm, it's very easy to use. Example code for downloading with requests library: from tqdm import tqdm import requests url = "https://proof.ovh.net/files/10Mb.dat" filepath = "test.dat" # Streaming, so we can iterate over the response. response = requests.get(url, stream=True) # Sizes in bytes. total_size = int(response.headers.get("content-length", 0)) block_size = 1024 with tqdm(total=total_size, unit="B", unit_scale=True) as progress_bar: with open(filepath, "wb") as file: for data in response.iter_content(block_size): progress_bar.update(len(data)) file.write(data) if total_size != 0 and progress_bar.n != total_size: raise RuntimeError("Could not download file")


Instruction:
How can we use Selenium Webdriver in colab.research.google.com?

I want to use Selenium Webdriver of Chrome in colab.research.google.com for fast processing. I was able to install Selenium using !pip install selenium but the webdriver of chrome needs a path to webdriverChrome.exe. How am I suppose to use it? P.S.- colab.research.google.com is an online platform which provides GPU for fast computational problems related to deep learning. Please refrain from solutions such as webdriver.Chrome(path).

Output: Recently Google collab was upgraded and since Ubuntu 20.04+ no longer distributes chromium-browser outside of a snap package, you can install a compatible version from the Debian buster repository: %%shell # Ubuntu no longer distributes chromium-browser outside of snap # # Proposed solution: https://askubuntu.com/questions/1204571/how-to-install-chromium-without-snap # Add debian buster cat > /etc/apt/sources.list.d/debian.list <<'EOF' deb [arch=amd64 signed-by=/usr/share/keyrings/debian-buster.gpg] http://deb.debian.org/debian buster main deb [arch=amd64 signed-by=/usr/share/keyrings/debian-buster-updates.gpg] http://deb.debian.org/debian buster-updates main deb [arch=amd64 signed-by=/usr/share/keyrings/debian-security-buster.gpg] http://deb.debian.org/debian-security buster/updates main EOF # Add keys apt-key adv --keyserver keyserver.ubuntu.com --recv-keys DCC9EFBF77E11517 apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 648ACFD622F3D138 apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 112695A0E562B32A apt-key export 77E11517 | gpg --dearmour -o /usr/share/keyrings/debian-buster.gpg apt-key export 22F3D138 | gpg --dearmour -o /usr/share/keyrings/debian-buster-updates.gpg apt-key export E562B32A | gpg --dearmour -o /usr/share/keyrings/debian-security-buster.gpg # Prefer debian repo for chromium* packages only # Note the double-blank lines between entries cat > /etc/apt/preferences.d/chromium.pref << 'EOF' Package: * Pin: release a=eoan Pin-Priority: 500 Package: * Pin: origin "deb.debian.org" Pin-Priority: 300 Package: chromium* Pin: origin "deb.debian.org" Pin-Priority: 700 EOF # Install chromium and chromium-driver apt-get update apt-get install chromium chromium-driver # Install selenium pip install selenium Then you can run selenium like this: from selenium import webdriver chrome_options = webdriver.ChromeOptions() chrome_options.add_argument('--headless') chrome_options.add_argument('--no-sandbox') chrome_options.headless = True wd = webdriver.Chrome('chromedriver',options=chrome_options) wd.get("https://www.webite-url.com")


Instruction:
Pandas - Slice large dataframe into chunks

I have a large dataframe (>3MM rows) that I'm trying to pass through a function (the one below is largely simplified), and I keep getting a Memory Error message. I think I'm passing too large of a dataframe into the function, so I'm trying to: 1) Slice the dataframe into smaller chunks (preferably sliced by AcctName) 2) Pass the dataframe into the function 3) Concatenate the dataframes back into one large dataframe def trans_times_2(df): df['Double_Transaction'] = df['Transaction'] * 2 large_df AcctName Timestamp Transaction ABC 12/1 12.12 ABC 12/2 20.89 ABC 12/3 51.93 DEF 12/2 13.12 DEF 12/8 9.93 DEF 12/9 92.09 GHI 12/1 14.33 GHI 12/6 21.99 GHI 12/12 98.81 I know that my function works properly, since it will work on a smaller dataframe (e.g. 40,000 rows). I tried the following, but I was unsuccessful with concatenating the small dataframes back into one large dataframe. def split_df(df): new_df = [] AcctNames = df.AcctName.unique() DataFrameDict = {elem: pd.DataFrame for elem in AcctNames} key_list = [k for k in DataFrameDict.keys()] new_df = [] for key in DataFrameDict.keys(): DataFrameDict[key] = df[:][df.AcctNames == key] trans_times_2(DataFrameDict[key]) rejoined_df = pd.concat(new_df) How I envision the dataframes being split: df1 AcctName Timestamp Transaction Double_Transaction ABC 12/1 12.12 24.24 ABC 12/2 20.89 41.78 ABC 12/3 51.93 103.86 df2 AcctName Timestamp Transaction Double_Transaction DEF 12/2 13.12 26.24 DEF 12/8 9.93 19.86 DEF 12/9 92.09 184.18 df3 AcctName Timestamp Transaction Double_Transaction GHI 12/1 14.33 28.66 GHI 12/6 21.99 43.98 GHI 12/12 98.81 197.62

Output: You can use list comprehension to split your dataframe into smaller dataframes contained in a list. n = 200000 #chunk row size list_df = [df[i:i+n] for i in range(0,df.shape[0],n)] Or use numpy array_split, list_df = np.array_split(df, math.ceil(len(df)/n)) You can access the chunks with: list_df[0] list_df[1] etc... Then you can assemble it back into a one dataframe using pd.concat. By AcctName list_df = [] for n,g in df.groupby('AcctName'): list_df.append(g)


Instruction:
How to launch python Idle from a virtual environment (virtualenv)

I have a package that I installed from a virtual environment. If I just launch the python interpreter, that package can be imported just fine. However, if I launch Idle, that package cannot be imported (since it's only available in one particular virtualenv and not global). How can I launch Idle from a virtualenv, so that all packages from the virtualenv would be available?

Output: For Python 3.6+, please see Paul Wicking's answer below. In Python prior to 3.6, IDLE is essentially from idlelib.PyShell import main if __name__ == '__main__': main() So you can launch it yourself unless you built the virtualenv without default packages.


Instruction:
SQLAlchemy: How do you delete multiple rows without querying

I have a table that has millions of rows. I want to delete multiple rows via an in clause. However, using the code: session.query(Users).filter(Users.id.in_(subquery....)).delete() The above code will query the results, and then execute the delete. I don't want to do that. I want speed. I want to be able to execute (yes I know about the session.execute):Delete from users where id in () So the Question: How can I get the best of two worlds, using the ORM? Can I do the delete without hard coding the query?

Output: Yep! You can call delete() on the table object with an associated where clause. Something like this: stmt = Users.__table__.delete().where(Users.id.in_(subquery...)) (and then don't forget to execute the statement: engine.execute(stmt)) source


Instruction:
How to drop a specific column of csv file while reading it using pandas?

I need to remove a column with label name at the time of loading a csv using pandas. I am reading csv as follows and want to add parameters inside it to do so. Thanks. pd.read_csv("sample.csv") I know this to do after reading csv: df.drop('name', axis=1)

Output: If you know the column names prior, you can do it by setting usecols parameter When you know which columns to use Suppose you have csv file with columns ["id", "name", "last_name"] and you want just ["name", "last_name"]. You can do it as below: import pandas as pd df = pd.read_csv("sample.csv", usecols=["name", "last_name"]) when you want first N columns If you don't know the column names but you want first N columns from dataframe. You can do it by import pandas as pd df = pd.read_csv("sample.csv", usecols=[i for i in range(n)]) Edit When you know name of the column to be dropped # Read column names from file cols = list(pd.read_csv("sample_data.csv", nrows=1)) print(cols) # Use list comprehension to remove the unwanted column in **usecol** df= pd.read_csv("sample_data.csv", usecols =[i for i in cols if i != "name"])


Instruction:
Convert XML/HTML Entities into Unicode String in Python

I'm doing some web scraping and sites frequently use HTML entities to represent non ascii characters. Does Python have a utility that takes a string with HTML entities and returns a unicode type? For example: I get back: ǎ which represents an "ǎ" with a tone mark. In binary, this is represented as the 16 bit 01ce. I want to convert the html entity into the value u'\u01ce'

Output: The standard lib’s very own HTMLParser has an undocumented function unescape() which does exactly what you think it does: up to Python 3.4: import HTMLParser h = HTMLParser.HTMLParser() h.unescape('© 2010') # u'\xa9 2010' h.unescape('© 2010') # u'\xa9 2010' Python 3.4+: import html html.unescape('© 2010') # u'\xa9 2010' html.unescape('© 2010') # u'\xa9 2010'


Instruction:
How to force migrations to a DB if some tables already exist in Django?

I have a Python/Django proyect. Due to some rolls back, and other mixed stuff we ended up in a kind of odd scenario. The current scenario is like this: DB has the correct tables DB can't be rolled back or dropped Code is up to date Migrations folder is behind the DB by one or two migrations. (These migrations were applied from somewhere else and that "somewhere else" doesn't exist anymore) I add and alter some models I run makemigrations New migrations are created, but it's a mix of new tables and some tables that already exist in the DB. If I run migrate it will complain that some of the tables that I'm trying to create already exist. What I need: To be able to run the migrations and kind of "ignore" the existing tables and apply the new ones. Or any alternative way to achieve this. Is that possible?

Output: When you apply a migration, Django inserts a row in a table called django_migrations. That's the only way Django knows which migrations have been applied already and which have not. So the rows in that table have to match the files in your migrations directory. If you've lost the migration files after they were applied, or done anything else to get things out of sync, you'll have problems.. because the migration numbers in your database refer to different migration files than the ones in your project. So before you do anything else, you need to bring things back into sync by deleting the django_migrations table rows for any migration files that you've lost somehow and can't get back. The table should contain rows for only those migrations that you do have and that were actually applied to the database correctly. Now you need to deal with any changes in your database that Django Migrations doesn't know about.. and for that there are a few options: If things worked out such that the database changes that were already applied to the database are in different migration files than the ones that weren't, then you can fix it by running your migrations one at a time using the --fake option on any changes that are in reality already in the database. The fake option just writes the row to the django_migrations table marking the migration as done. Only do this if the database does in fact already have all the changes contained in that migration file. And those migration files that contain only changes which have not been applied to the database, run without the --fake option and Django will apply them. eg: # database already has it manage.py migrate myapp 0003 --fake # need it manage.py migrate myapp 0004 # database already has it manage.py migrate myapp 0005 --fake If you have migration files where some but not all of the changes have been applied, then you have a bigger problem. In that case, there are several ways to go about it (choose ONLY ONE): Edit the migration files to put changes that have already been applied (whether Django did it or you did it manually does not matter) into lower number migrations, and put everything you need done into higher numbered files. Now you can --fake the lower number ones, and run the higher numbered ones as normal. Let's say you have 10 changes you made to your models, and 5 of those changes are actually in the database already, but Django doesn't know about them.. so when you run makemigrations, a new migration is created with all 10 changes. This will normally fail because the database server can't for example add a column that already exists. Move these already-applied changes out of your new migration file, into the previous (already applied) migration file. Django will then assume that these were applied with the previous migration and will not try to apply them again. You can then migrate as normal and the new changes will be applied. If you don't want to touch your older migration file, a cleaner way to do this is to first run makemigrations --empty appname to create an empty migration file. Then run makemigrations which will create another migration with all the changes that Django thinks need to be done. Move the already done migrations from that file into the empty migration you created.. then --fake that one. This will put Django's understanding of what the database looks like will be in sync with reality and you can then migrate as normal, applying the changes in the last migration file. Get rid of any new migrations you just created using makemigrations. Now, comment out or put back anything in your models that has not been applied to the database, leaving your code matching what's actually in the database. Now you can do makemigrations and migrate appname --fake and you will get things back in sync. Then uncomment your new code and run 'makemigrations' then migrate as normal and the changes will be applied. If the changes are small (for example, adding a few fields), sometimes this is easiest. If the changes are large, it isn't.... You can go ahead and (carefully) make the database changes yourself, bringing the database up to date. Now just run migrate --fake and if you didn't mess up then everything will be ok. Again, this is easy for smaller changes, not as easy for complicated ones. You can run manage.py sqlmigrate > mychanges.sql. This generates mychanges.sql containing all the SQL Django WOULD have executed against the database. Now edit that file to remove any changes that have already been applied, leaving what needs to be done. Execute that SQL using pgadmin or psql (you're using postgresql I hope). Now the changes have all been made.. so you can run manage.py migrate --fake, this will bring Django into sync with reality and you should be all set. If your SQL skills are sufficient, this is probably the most straightforward solution. I should add two warnings: First, if you apply a later migration, eg 0003_foobar.py, and then things don't work out and you decide to try going back and apply 0002_bazbuz.py, then Django will TAKE STUFF OUT OF YOUR DATABASE. For example a column you might have added in 0003 will be dropped along with its data. Since you say you can't lose data, be very careful about going back. Second, do not rush into running --fake migrations. Make sure that the entire migration you are about to fake is actually in the database already. Else it gets very confusing. If you do regret faking migrations and don't want to roll back, you can erase django's knowledge of the faked migration by deleting that row from the django_migrations table. It is ok to do this.. if you understand what you are doing. If you know that the migration really was not applied, then it's ok.


Instruction:
How do I find an image contained within an image?

I'm currently building what basically amounts to a cross between a search engine and a gallery for web comics that's focused on citing sources and giving authors credit. I'm trying to figure out a way to search an image to find characters within it. For example: Assuming I have the red character and the green character saved as Red Man and Green Man how do I determine if an image contains one or the other. This doesn't need to have 100% recognition or anything is this is more of an added feature I'd like to create, I'm just not sure where to start. I've done a lot of googling for image recognition but haven't found much helpful. For what it's worth, I'd prefer to do this using Python.

Output: For anyone who stumbles across this in the future. This can be done with template matching. To summarize (my understanding), template matching looks for an exact match of one image within another image. Here's an example of how to do it within Python: import cv2 method = cv2.TM_SQDIFF_NORMED # Read the images from the file small_image = cv2.imread('small_image.png') large_image = cv2.imread('large_image.jpeg') result = cv2.matchTemplate(small_image, large_image, method) # We want the minimum squared difference mn,_,mnLoc,_ = cv2.minMaxLoc(result) # Draw the rectangle: # Extract the coordinates of our best match MPx,MPy = mnLoc # Step 2: Get the size of the template. This is the same size as the match. trows,tcols = small_image.shape[:2] # Step 3: Draw the rectangle on large_image cv2.rectangle(large_image, (MPx,MPy),(MPx+tcols,MPy+trows),(0,0,255),2) # Display the original image with the rectangle around the match. cv2.imshow('output',large_image) # The image is only displayed if we call this cv2.waitKey(0)


Instruction:
Visual Studio Code does not detect Virtual Environments

Visual Studio Code does not detect virtual environments. I run vscode in the folder where the venv folder is located, when I try to select the kernel in vscode I can see the main environment and one located elsewhere on the disk. Jupyter running in vscode also doesn't see this environment. I have installed ipykernel in this environment. I tried to reinstall vscode and python extension. I tried to set the path in settings.json inside .vscode: { "python.pythonPath": ".\\venv\\Scripts\\python.exe" } Windows 10 Python 3.6.7 (64-bit) VSCode 1.54.3

Output: In VSCode open your command palette — Ctrl+Shift+P by default Look for Python: Select Interpreter In Select Interpreter choose Enter interpreter path... and then Find... Navigate to your venv folder — eg, ~/pyenvs/myenv/ or \Users\Foo\Bar\PyEnvs\MyEnv\ In the virtual environment folder choose <your-venv-name>/bin/python or <your-venv-name>/bin/python3 The issue is that VSCode's Python extension by default uses the main python or python3 program while venv effectively creates a "new" python/python3 executable (that is kind of the point of venv) so the extension does not have access to anything (available modules, namespaces, etc) that you have installed through a venv since the venv specific installations are not available to the main Python interpreter (again, this is by design—like how applications installed in a VM are not available to the host OS)


Instruction:
How to tell flake8 to ignore comments

I'm using flake8 in emacs in order to clean up my python code. I find it annoying to have my comments flagged as errors (E501 line too long (x > 79 characters)). I'm wondering if anyone knows a way to kindly ask flake8 to ignore comments, both single and multi-line, but still let me know when my non-comment lines are too long? Thanks in advance!

Output: I've figured out a possible solution to this, but there might be something better. If you write a comment that will raise an E501 error, i.e. it is too long, you can append that line with # noqa: E501, and flake8 will ignore it. For example: # This is a really really long comment that would usually be flagged by flake8 because it is longer than 79 characters would usually raise an E501, but # This is a really really long comment that would usually be flagged by flake8 because it is longer than 79 characters # noqa: E501 will not. documented here.


Instruction:
return max value from pandas dataframe as a whole, not based on column or rows

I am trying to get the max value from a panda dataframe as a whole. I am not interested in what row or column it came from. I am just interested in a single max value within the DataFrame. Here is my DataFrame: df = pd.DataFrame({'group1': ['a','a','a','b','b','b','c','c','d','d','d','d','d'], 'group2': ['c','c','d','d','d','e','f','f','e','d','d','d','e'], 'value1': [1.1,2,3,4,5,6,7,8,9,1,2,3,4], 'value2': [7.1,8,9,10,11,12,43,12,34,5,6,2,3]}) This is what it looks like: group1 group2 value1 value2 0 a c 1.1 7.1 1 a c 2.0 8.0 2 a d 3.0 9.0 3 b d 4.0 10.0 4 b d 5.0 11.0 5 b e 6.0 12.0 6 c f 7.0 43.0 7 c f 8.0 12.0 8 d e 9.0 34.0 9 d d 1.0 5.0 10 d d 2.0 6.0 11 d d 3.0 2.0 12 d e 4.0 3.0 Expected output: 43.0 I was under the assumption that df.max() would do this job but it returns a max value for each column but I am not interested in that. I need the max from an entire dataframe.

Output: The max of all the values in the DataFrame can be obtained using df.to_numpy().max(), or for pandas < 0.24.0 we use df.values.max(): In [10]: df.to_numpy().max() Out[10]: 'f' The max is f rather than 43.0 since, in CPython2, In [11]: 'f' > 43.0 Out[11]: True In CPython2, Objects of different types ... are ordered by their type names. So any str compares as greater than any int since 'str' > 'int'. In Python3, comparison of strings and ints raises a TypeError. To find the max value in the numeric columns only, use df.select_dtypes(include=[np.number]).max()


Instruction:
How to sort a list with two keys but one in reverse order?

I was wondering what would be a Pythonic way of sorting a list of tuples by two keys whereby sorting with one (and only one) key would be in a reverse order and sorting with the the other would be case insensitive. More specifically, I have a list containing tuples like: myList = [(ele1A, ele2A),(ele1B, ele2B),(ele1C, ele2C)] I can use the following code to sort it with two keys: sortedList = sorted(myList, key = lambda y: (y[0].lower(), y[1])) To sort in reverse order I can use sortedList = sorted(myList, key = lambda y: (y[0].lower(), y[1]), reverse = True) But this would sort in a reverse order with two keys.

Output: Two keys will be used when we need to sort a list with two constraints: one in ascending order and the other in descending, in the same list or any In your example, sortedList = sorted(myList, key = lambda y: (y[0].lower(), y[1])) you can sort entire list only in one order. You can try these and check what's happening: sortedList = sorted(myList, key = lambda y: (y[0].lower(), -y[1])) sortedList = sorted(myList, key = lambda y: (-y[0].lower(), y[1])) sortedList = sorted(myList, key = lambda y: (-y[0].lower(), -y[1]))


Instruction:
SQLAlchemy or psycopg2?

I am writing a quick and dirty script which requires interaction with a database (PG). The script is a pragmatic, tactical solution to an existing problem. however, I envisage that the script will evolve over time into a more "refined" system. Given the fact that it is currently being put together very quickly (i.e. I don't have the time to pour over huge reams of documentation), I am tempted to go the quick and dirty route, using psycopg. The advantages for psycopg2 (as I currently understand it) is that: written in C, so faster than sqlAlchemy (written in Python)? No abstraction layer over the DBAPI since works with one db and one db only (implication -> fast) (For now), I don't need an ORM, so I can directly execute my SQL statements without having to learn a new ORM syntax (i.e. lightweight) Disadvantages: I KNOW that I will want an ORM further down the line psycopg2 is ("dated"?) - don't know how long it will remain around for Are my perceptions of SqlAlchemy (slow/interpreted, bloated, steep learning curve) true - IS there anyway I can use sqlAlchemy in the "rough and ready" way I want to use psycopg - namely: execute SQL statements directly without having to mess about with the ORM layer, etc. Any examples of doing this available?

Output: SQLAlchemy is a ORM, psycopg2 is a database driver. These are completely different things: SQLAlchemy generates SQL statements and psycopg2 sends SQL statements to the database. SQLAlchemy depends on psycopg2 or other database drivers to communicate with the database! As a rather complex software layer SQLAlchemy does add some overhead but it also is a huge boost to development speed, at least once you learned the library. SQLAlchemy is an excellent library and will teach you the whole ORM concept, but if you don't want to generate SQL statements to begin with then you don't want SQLAlchemy.


Instruction:
Python try except block inside lambda

Is it possible to use a try except block inside of a lambda function? I need the lambda function to convert a certain variable into an integer, but not all of the values will be able to be converted into integers.

Output: Nope. A Python lambda can only be a single expression. Use a named function. It is convenient to write a generic function for converting types: def tryconvert(value, default, *types): for t in types: try: return t(value) except (ValueError, TypeError): continue return default Then you can write your lambda: lambda v: tryconvert(v, 0, int) You could also write tryconvert() so it returns a function that takes the value to be converted; then you don't need the lambda: def tryconvert(default, *types): def convert(value): for t in types: try: return t(value) except (ValueError, TypeError): continue return default # set name of conversion function to something more useful namext = ("_%s_" % default) + "_".join(t.__name__ for t in types) if hasattr(convert, "__qualname__"): convert.__qualname__ += namext convert.__name__ += namext return convert Now tryconvert(0, int) returns a function convert_0_int that takes a value and converts it to an integer, and returns 0 if this can't be done. You can use this function right away (not saving a copy): mynumber = tryconert(0, int)(value) Or save it to call it later: intconvert = tryconvert(0, int) # later... mynumber = intconvert(value)


Instruction:
Elegant way to make all dirs in a path

Here are four paths: p1=r'\foo\bar\foobar.txt' p2=r'\foo\bar\foo\foo\foobar.txt' p3=r'\foo\bar\foo\foo2\foobar.txt' p4=r'\foo2\bar\foo\foo\foobar.txt' The directories may or may not exist on a drive. What would be the most elegant way to create the directories in each path? I was thinking about using os.path.split() in a loop, and checking for a dir with os.path.exists, but I don't know it there's a better approach.

Output: You are looking for os.makedirs() which does exactly what you need. The documentation states: Recursive directory creation function. Like mkdir(), but makes all intermediate-level directories needed to contain the leaf directory. Raises an error exception if the leaf directory already exists or cannot be created. By default it fails if the leaf directory already exists; you'll want to test for existence before calling os.makedirs() or use os.makedirs(..., exist_ok=True) to ignore existence.


Instruction:
Python replace string pattern with output of function

I have a string in Python, say The quick @red fox jumps over the @lame brown dog. I'm trying to replace each of the words that begin with @ with the output of a function that takes the word as an argument. def my_replace(match): return match + str(match.index('e')) #Psuedo-code string = "The quick @red fox jumps over the @lame brown dog." string.replace('@%match', my_replace(match)) # Result "The quick @red2 fox jumps over the @lame4 brown dog." Is there a clever way to do this?

Output: You can pass a function to re.sub. The function will receive a match object as the argument, use .group() to extract the match as a string. >>> def my_replace(match): ... match = match.group() ... return match + str(match.index('e')) ... >>> string = "The quick @red fox jumps over the @lame brown dog." >>> re.sub(r'@\w+', my_replace, string) 'The quick @red2 fox jumps over the @lame4 brown dog.'