When a new room was being added to his house, he found the "eating habits" of some of the workers "strange" as "they were eating only 'poha' (flattened rice)", the 63-year-old BJP leader said.
"I suspected these workers were residents of Bangladesh. Two days after I became suspicious, they stopped working at my house," Mr Vijayvargiya told reporters. "I have not filed any police complaint yet. I only mentioned this incident to warn people."
The BJP leader's comments seemed unpalatable to many on Twitter who clapped back with jokes.
Many found his comment puzzling as Indore is famous for its poha, a popular breakfast item.
The remarks also led to a number of Twitter users sharing pictures of poha on the microblogging platform.
Twitter thanks Kailash Vijayvargiya as #Poha is trending
While speaking at the event, Mr Vijayvargiya also claimed that a Bangladeshi terrorist was keeping a watch on him for the last one and a half years.
"Whenever I go out, six armed security personnel follow me. What is happening in this country? Will outside people enter and spread so much terror? The CAA is in the interest of the country," he said.