We investigate theoretically a hybrid system consisting of a Bose-Einstein condensate (BEC) trapped inside a laser-driven membrane-in-the-middle optomechanical cavity assisted with squeezed vacuum injection whose moving membrane interacts both linearly and quadratically with the radiation pressure of the cavity. It is shown that such a hybrid system is very suitable for generating strong quadrature squeezing in the mechanical mode of the membrane and the Bogoliubov mode of the BEC in the unresolved sideband regime. More interestingly, by choosing a suitable sign for the quadratic optomechanical coupling (QOC), one can achieve a very high degree of squeezing in the mechanical mode and a strong entanglement between the mechanical and atomic modes without the necessity of using squeezed light injection. Furthermore, the QOC changes the effective oscillation frequencies of both the mechanical and the atomic modes and affects their relaxation times. It can also make the system switch from optical bistability to tristability.