There are many theories on how leia was caught by jabba. The movie only shows her being caught. Im going to tell you how jabba found out that she was infiltrating his palace.
Leia first came to jabba's palace the night before she was captured. although she never went inside, she could tell it was scary in there. She found one of jabba's guards outside and made a deal with him. Stupidly, and without thinking, she paid the guard in advance. all she wanted him to do was to open the front gate when han, chewie, and her escape the next night. he agreed and took the money. This guard however was not an honorable person and planned to betray her. the next day he did exactly what he promissed. he let Bouchh, aka leia, and chewie into the palace. he had totally fooled them. Now if you've seen RETURN OF THE JEDI you know what leia does. She pretends to be a bounty hunter and "sells" chewie to the slug crime lord. She is then welcomed into the party. Now jabba, at that time, did not know that he had been duped, but he was about to find out.
That night when leia had already been showned to her room, the guard that leia had paid, paid a visit to jabba and told him her plan. "How dare that princess try to take away my prize possesion." Jabba screamed
"Should I go get her." the guard asked
"No", said Jabba,"Im going to let her set han free and let her think that she had won. then Im going the to pulled the carpet out from under her and dash her hopes. I'll put solo into custity."
"What are you gonna do to her" asked the guard.
"Im going to make her my new slave girl and punish her for setting solo free."
So, jabba waited and told everybody that was still in the thone room to pretend to sleep until he gave them the signal. when someone asked him what the signal was going to be. He answerd them with a thunderous laugh.
later that night leia prepared to do something very dangerous. She looked in the mirror that was in the room that was assigned to and told herself that it was going to be fine. little did she know that she was falling into a trap. She began to walk down the steps, trying to make as little noise as possible. Suddenly, she found the thrown room. As leia walks into the room jabba and most of his men are watching from behind a few curtains. Scilently, Jabba watches carefully as she sets han solo free. When Leia takes off her mask and kisses Han Jabba is very aroused. the gurd told him that she was pritty but jabba was stunned by her beauty. As she began to pick Han up Jabba gave out a loud laugh. Leia soon realized who it was and shivered at the sound of him.