|author||Nadav Amit <email@example.com>||2017-08-10 15:23:59 -0700|
|committer||Linus Torvalds <firstname.lastname@example.org>||2017-08-10 15:54:07 -0700|
mm: migrate: fix barriers around tlb_flush_pending
Reading tlb_flush_pending while the page-table lock is taken does not require a barrier, since the lock/unlock already acts as a barrier. Removing the barrier in mm_tlb_flush_pending() to address this issue. However, migrate_misplaced_transhuge_page() calls mm_tlb_flush_pending() while the page-table lock is already released, which may present a problem on architectures with weak memory model (PPC). To deal with this case, a new parameter is added to mm_tlb_flush_pending() to indicate if it is read without the page-table lock taken, and calling smp_mb__after_unlock_lock() in this case. Link: http://email@example.com Signed-off-by: Nadav Amit <firstname.lastname@example.org> Acked-by: Rik van Riel <email@example.com> Cc: Minchan Kim <firstname.lastname@example.org> Cc: Sergey Senozhatsky <email@example.com> Cc: Andy Lutomirski <firstname.lastname@example.org> Cc: Mel Gorman <email@example.com> Cc: "David S. Miller" <firstname.lastname@example.org> Cc: Andrea Arcangeli <email@example.com> Cc: Heiko Carstens <firstname.lastname@example.org> Cc: Hugh Dickins <email@example.com> Cc: Ingo Molnar <firstname.lastname@example.org> Cc: Jeff Dike <email@example.com> Cc: Martin Schwidefsky <firstname.lastname@example.org> Cc: Mel Gorman <email@example.com> Cc: Nadav Amit <firstname.lastname@example.org> Cc: Russell King <email@example.com> Cc: Tony Luck <firstname.lastname@example.org> Cc: Yoshinori Sato <email@example.com> Signed-off-by: Andrew Morton <firstname.lastname@example.org> Signed-off-by: Linus Torvalds <email@example.com>
1 files changed, 10 insertions, 4 deletions
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index f58f76ee1dfa..0e478ebd2706 100644
@@ -526,12 +526,12 @@ static inline cpumask_t *mm_cpumask(struct mm_struct *mm)
* Memory barriers to keep this state in sync are graciously provided by
* the page table locks, outside of which no page table modifications happen.
- * The barriers below prevent the compiler from re-ordering the instructions
- * around the memory barriers that are already present in the code.
+ * The barriers are used to ensure the order between tlb_flush_pending updates,
+ * which happen while the lock is not taken, and the PTE updates, which happen
+ * while the lock is taken, are serialized.
static inline bool mm_tlb_flush_pending(struct mm_struct *mm)
return atomic_read(&mm->tlb_flush_pending) > 0;
@@ -554,7 +554,13 @@ static inline void inc_tlb_flush_pending(struct mm_struct *mm)
/* Clearing is done after a TLB flush, which also provides a barrier. */
static inline void dec_tlb_flush_pending(struct mm_struct *mm)
+ * Guarantee that the tlb_flush_pending does not not leak into the
+ * critical section, since we must order the PTE change and changes to
+ * the pending TLB flush indication. We could have relied on TLB flush
+ * as a memory barrier, but this behavior is not clearly documented.