# Get embedding from finetuned BertForSequenceClassification model

**URL:** <https://discuss.huggingface.co/t/get-embedding-from-finetuned-bertforsequenceclassification-model/14897>\
**Category:** 🤗Transformers\
**Created:** [February 19, 2022, 5:32pm UTC](https://discuss.huggingface.co/t/get-embedding-from-finetuned-bertforsequenceclassification-model/14897 "2022-02-19T17:32:17Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![jackkwok](https://avatars.discourse-cdn.com/v4/letter/j/ecae2f/32.png) [@jackkwok](https://discuss.huggingface.co/u/jackkwok)\
**Post date:** [February 19, 2022, 5:32pm UTC](https://discuss.huggingface.co/t/get-embedding-from-finetuned-bertforsequenceclassification-model/14897/1 "2022-02-19T17:32:17Z")

</div>

How do I get the embedding tensors from my fine-tuned `BertForSequenceClassification` model? The issue is ‘BertForSequenceClassification’ object has no attribute ‘embeddings’.

However, I can successfully get the BERT embedding for use in Tensorboard’s Embedding Projector tool with this code:

```auto
import transformers
from transformers import AutoTokenizer, AutoModelForSequenceClassification, AutoConfig
from torch.utils.tensorboard import SummaryWriter

import tensorflow as tf
import tensorboard as tb

# https://github.com/pytorch/pytorch/issues/30966#issuecomment-582747929
tf.io.gfile = tb.compat.tensorflow_stub.io.gfile

writer = SummaryWriter(log_dir='tensorboard_embedding_custom2')

model = transformers.BertModel.from_pretrained('bert-base-uncased')

tokenizer = transformers.BertTokenizer.from_pretrained('bert-base-uncased')
words = tokenizer.vocab.keys()
word_embedding = model.embeddings.word_embeddings.weight
writer.add_embedding(word_embedding,
                     metadata = words,
                     tag = f'word embedding')

writer.flush()
writer.close()

```

---

<div class="post-metadata">

**Author:** ![beneyal](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/beneyal/32/27988_2.png) [@beneyal](https://discuss.huggingface.co/u/beneyal)\
**Post date:** [February 19, 2022, 5:55pm UTC](https://discuss.huggingface.co/t/get-embedding-from-finetuned-bertforsequenceclassification-model/14897/2 "2022-02-19T17:55:41Z")

</div>

Hello! 👋

For `BertForSequenceClassification` you can do almost exactly what you did with `BertModel`:

```python
>>> model = BertForSequenceClassification.from_pretrained("bert-base-uncased")
>>> model.bert.embeddings.word_embeddings.weight
Parameter containing:
tensor([[-0.0102, -0.0615, -0.0265, ..., -0.0199, -0.0372, -0.0098],
        [-0.0117, -0.0600, -0.0323, ..., -0.0168, -0.0401, -0.0107],
        [-0.0198, -0.0627, -0.0326, ..., -0.0165, -0.0420, -0.0032],
        ...,
        [-0.0218, -0.0556, -0.0135, ..., -0.0043, -0.0151, -0.0249],
        [-0.0462, -0.0565, -0.0019, ..., 0.0157, -0.0139, -0.0095],
        [0.0015, -0.0821, -0.0160, ..., -0.0081, -0.0475, 0.0753]],
       requires_grad=True)

```