hanzceo/SimpleEmbed-dev1

SimpleEmbed

This is a sentence-transformers model trained on the hanzceo/sts-en-en, jinaai/negation-dataset, andersonbcdefg/jina_negation_v2, WhereIsAI/github-issue-similarity and aisuko/quora_duplicate_questions datasets. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Maximum Sequence Length: inf tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text
• Training Datasets:
  • hanzceo/sts-en-en
  • jinaai/negation-dataset
  • andersonbcdefg/jina_negation_v2
  • WhereIsAI/github-issue-similarity
  • aisuko/quora_duplicate_questions
• Language: en
• License: mit

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): StaticEmbedding({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("hanzceo/SimpleEmbed-dev1")
# Run inference
queries = [
    'Two men working on their knees in front of a building.',
]
documents = [
    'There are people working.',
    'There are people not working.',
    'A man is in a kayak.',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 1024] [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[ 0.4950,  0.1923, -0.0330]])

Training Details

Training Datasets

hanzceo/sts-en-en

hanzceo/sts-en-en

• Dataset: hanzceo/sts-en-en
• Size: 1,182 training samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 100 samples:

  	anchor	positive	negative
  type	string	string	string
  modality	text	text	text
  details	min: 60 characters mean: 79.23 characters max: 107 characters	min: 59 characters mean: 75.64 characters max: 112 characters	min: 41 characters mean: 72.0 characters max: 98 characters

• Samples:

  anchor	positive	negative
  The intricate framework elucidates previously uncharted dimensions of cellular plasticity	Researchers have unveiled an innovative approach that captures dynamic fluctuations within cellular architecture	The old wooden bridge creaked under the weight of the passing tractor
  Observers note a pronounced correlation between epigenetic markers and metabolic fluxes	Analysts discovered subtle shifts in transcriptional patterns across divergent cell populations	The stock market experienced a slight dip following the unexpected jobs report
  The novel assay integrates multi-omics datasets to reconstruct regulatory networks	Investigators applied machine learning to synthesize heterogeneous biological information	Apples float in water because they are made up of 25 percent air

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          1024,
          512,
          256,
          128,
          64,
          32
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

jinaai/negation-dataset

jinaai/negation-dataset

• Dataset: jinaai/negation-dataset
• Size: 10,000 training samples
• Columns: anchor, entailment, and negative
• Approximate statistics based on the first 100 samples:

  	anchor	entailment	negative
  type	string	string	string
  modality	text	text	text
  details	min: 25 characters mean: 68.11 characters max: 189 characters	min: 14 characters mean: 35.34 characters max: 107 characters	min: 16 characters mean: 38.9 characters max: 105 characters

• Samples:

  anchor	entailment	negative
  Two young girls are playing outside in a non-urban environment.	Two girls are playing outside.	Two girls are not playing outside.
  A man with a red shirt is watching another man who is standing on top of a attached cart filled to the top.	A man is standing on top of a cart.	A man is not standing on top of a cart.
  A man in a blue shirt driving a Segway type vehicle.	A person is riding a motorized vehicle.	A person is not riding a motorized vehicle.

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          1024,
          512,
          256,
          128,
          64,
          32
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

andersonbcdefg/jina_negation_v2

andersonbcdefg/jina_negation_v2

• Dataset: andersonbcdefg/jina_negation_v2
• Size: 50,000 training samples
• Columns: query, pos, and neg
• Approximate statistics based on the first 100 samples:

  	query	pos	neg
  type	string	string	string
  modality	text	text	text
  details	min: 33 characters mean: 103.12 characters max: 309 characters	min: 20 characters mean: 58.65 characters max: 164 characters	min: 22 characters mean: 58.88 characters max: 156 characters

• Samples:

  query	pos	neg
  A dog happily looking onward in the back seat of a car.	A dog sitting down inside of a vehicle.	A dog standing up outside of a vehicle.
  Alas, madame, said Poirot, "I thought you had come to honour me with a visit!"	Poirot said that he thought the woman had come to visit him.	Poirot said that he thought the woman had not come to visit him.
  yeah i i i agree i the thing that scares me uh though about where i would i would want definitely want some sort of legislation and coming from the north east i'm just not used to seeing um these and i i know this may sound kind of stereo typical but the cowboys with the gun racks in the back of their trucks	I would want the law to cover guns.	I would not want the law to cover guns.

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          1024,
          512,
          256,
          128,
          64,
          32
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

WhereIsAI/github-issue-similarity

WhereIsAI/github-issue-similarity

• Dataset: WhereIsAI/github-issue-similarity
• Size: 9,457 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 100 samples:

  	anchor	positive
  type	string	string
  modality	text	text
  details	min: 0 characters mean: 4993.44 characters max: 135575 characters	min: 0 characters mean: 3329.07 characters max: 19882 characters

• Samples:

  anchor	positive
  Bug summary I put a torch.Tensor in matplotlib.pyplot.hist() , but it draw a wrong graphic and take a long time. Although transform to numpy, the function work well. But all the others function I used are work well on tensor. So I think its a bug. Code for reproduction import matplotlib.pyplot as plt import torch plt.hist(torch.randn...	Bug report Bug summary Generating np.random.randn(1000) values, visualizing them with plt.hist(). Works fine with Numpy. When I replace Numpy with tensorflow.experimental.numpy, Matplotlib 3.3.4 fails to display the histogram correctly. Matplotlib 3.2.2 works fine. Code for reproduction import matplotlib.
  https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/container_bridge.go#L122-L143 container_bridge.go assumes that the virtual IP of services & pods will be in the 10. space. I propose there is no reason to make this assumption. As outlined in #15932, cluster admins may need to deploy to hosts in which 10. is reserved for the nodes. In such a case, Kubelets must support an alternative range.	Today kubelet sets up an iptables MASQUERADE rule for any traffic destined for anything except 10.0.0.0/8. This is close, but not even correct on GCE, and certainly not right elsewhere. First GCE. We probably want something like: iptables -t nat -N KUBE-IPMASQ iptables -t nat -A KUBE-IPMASQ -d 10.0.0.0/8 -j RETURN iptables -t nat -A KUBE-IPMASQ -d 172.16.0.0/12 -j RETURN iptables -t nat -A KUBE-IPMASQ -d 192.168.0.0/16 -j RETURN iptables -t nat -A KUBE-IPMASQ -j MASQUERADE iptables -t nat -I...
  Is there an existing issue for this? I have searched the existing issues This issue exists in the latest npm version I am using the latest npm Current Behavior Currently, my package.json specifies "typescript": "^5.0.2". When I change it to say "typescript": "^5.0.3", npm 9 spins for 4:28 before deciding it doesn't exist. For comparison, npm 8 installs it with no problem in 0:44. Ironically, I can't upgrade npm to 9.6 due to this issue: npm 9.5.1 times out when I run npm i -g npm. Expec...	Is there an existing issue for this? I have searched the existing issues This issue exists in the latest npm version I am using the latest npm Current Behavior When running npm install it will sometimes hang at a random point. When it does this, it is stuck forever. CTRL+C will do nothing the first time that combination is pressed when this has occurred. Pressing that key combination the second time will make the current line (the one showing the little progress bar) disappear but that's it. No further responses to that key combination are observed. The C...

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          1024,
          512,
          256,
          128,
          64,
          32
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

aisuko/quora_duplicate_questions

aisuko/quora_duplicate_questions

• Dataset: aisuko/quora_duplicate_questions at a14d279
• Size: 149,263 training samples
• Columns: question1 and question2
• Approximate statistics based on the first 100 samples:

  	question1	question2
  type	string	string
  modality	text	text
  details	min: 16 characters mean: 54.75 characters max: 139 characters	min: 21 characters mean: 54.34 characters max: 127 characters

• Samples:

  question1	question2
  Astrology: I am a Capricorn Sun Cap moon and cap rising...what does that say about me?	I'm a triple Capricorn (Sun, Moon and ascendant in Capricorn) What does this say about me?
  How can I be a good geologist?	What should I do to be a great geologist?
  How do I read and find my YouTube comments?	How can I see all my Youtube comments?

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          1024,
          512,
          256,
          128,
          64,
          32
      ],
      "matryoshka_weights": [
          1,
          1,
          1,
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 256
• learning_rate: 0.002
• weight_decay: 0.01
• num_train_epochs: 10
• warmup_ratio: 0.1
• seed: 69
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• prediction_loss_only: True
• per_device_train_batch_size: 256
• per_device_eval_batch_size: 8
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 0.002
• weight_decay: 0.01
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 10
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 69
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss
0.0012	1	7.9156
1.1601	1000	11.1122
2.3202	2000	6.1974
3.4803	3000	4.1201
4.6404	4000	3.2187
5.8005	5000	2.6037
6.9606	6000	2.2084
8.1206	7000	1.887
9.2807	8000	1.8408

Training Time

• Training: 37.3 minutes

Framework Versions

• Python: 3.12.6
• Sentence Transformers: 5.6.0
• Transformers: 4.56.0
• PyTorch: 2.8.0+cu129
• Accelerate: 1.10.1
• Datasets: 5.0.0
• Tokenizers: 0.22.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}